PJRC Website Speed & Reliability, Feedback Wanted?

Status
Not open for further replies.
PaulStoffregen

PaulStoffregen

Well-known member
This morning the PJRC website was offline for several minutes. It appears some errant bot tried to rapidly access forum pages, which placed far too much load on the server. Ultimately, the server had to be rebooted by the hosting service.

I'd like to ask, have you notice slowness or non-responding pages?

We're looking into options to improve the site's hosting....
 
My first round of suggestions for the ADC improvements were wiped out. Had to do with the auto-logout I usually experience here with longer posts. But usually, the post is saved and then posted after I login again. Not this time. Oh Well.
 
Last edited:
Looks like we had another slowdown this morning, around 3:45 am (pacific time), lasting about 4 minutes, and another close to 4am, also about 3-4 minutes.

From the server log file, it looks like an overly aggressive bot. The pattern seems to involve rapidly fetching several dozen random pages. It's playing tricks like using a different browser name in the user agent field on each request. The pages it accesses are a mix of random forum pages, plus URLs from the main site (which return 404 errors to the bot, because it's using "forum.pjrc.com" on URLs that should begin with "www.pjrc.com"). After several pages, it always goes to the forum login page and makes a few attempts. Then it repeats the process, hitting many random pages, then making more login attempts.

Damn spammers!

At least the good news is we're not getting any significant amount of spam on the forum anymore, so obviously they're not managing to get in. :)
 
I don't get it. What's the probability of getting money from a spam-ad here? 0.00000001?
Or is it just juvenile IT vandalism?
 
nox771 said:
Just FYI - was down again today at 11am CST.
Click to expand...

The site was just down for a while or being DDOS'd
Click to expand...

Thanks. Looks like there were 3 separate bot attacks this morning around that time.

Sadly, it's looking like our little server is perfectly capable of handling normal forum usage where several real humans do things at normal human speeds. But as soon as 1 bot starts rapidly pounding the forum, the load is too great for the website to keep up. It is a dedicated server, but it's an old machine (single core Pentium 4) with only 1 GB of RAM. We'd never survive a DDOS.....

It's looking like we really need to buy a 2nd much faster server just for the forum. Our old server just isn't fast enough to respond rapidly enough (with the substantial overhead this forum takes) for an anti-bot detection to reliably tell the difference from a human who just happens to click several pages quickly.

I've been looking into hosting companies since last night. My hope is to get a server dedicated to the forum online and the forum moved over within the next few days.
 
I'm sure you've thought about black-listing the offending remote host IP address within your router. Mine has such a black list. I have most of China in it.

For a year+ I've been using a Windows 7 Enterprise VM from Scale Matrix in San Diego. Month-to-month pricing - not the cheapest. They charge less for Linux. $100-200/mo with do-your-own backups. Much more if you want an SLA in the contract.
Amazon EC2 is low cost but hard to use.
Rackspace essentially resells Amazon S3 and EC2.
I'm going to change to use a provider that doesn't use the high cost, complex VMware approach.
A problem with all these kinds of VMs is that if/when you need to reboot the VM and boot into a different OS/program, such as if you have to boot a recovery standalone program... you have to pay them to do so via the VM admin which you can't access.

One that I like uses a KVM OVER IP, so when you reboot, you keep total control via KVM, and you're not dependent on the VM manager GUI being available to YOU, remotely. I mean a physical KVM so it matters not what OS or standalone program is running on the VM. Such as a DVD image file that has a program for disk recovery for when you lose the OS due to hacking or due to a failure of the provider's RAID (which happened once to me - my own backup saved my behind).
 
In case anyone's curious, the new forum server will be a E3-1230 Xeon with 16 GB RAM and a 80 GB SSD. That's a pretty big step up from the P4 with 1GB RAM and 7200 rpm drive we're on now. Hopefully the SSD will let mysql respond many more queries/sec and we can turn on PHP opcode caching and tune mysql better with the extra RAM.

I'm looking into bandwidth limiting options. But I don't want any bandwidth limiting to kick in for real humans, even if they click pretty fast. Anyone have any experience with such things?
 
PaulStoffregen said:
This morning the PJRC website was offline for several minutes. It appears some errant bot tried to rapidly access forum pages, which placed far too much load on the server. Ultimately, the server had to be rebooted by the hosting service.

I'd like to ask, have you notice slowness or non-responding pages?

We're looking into options to improve the site's hosting....
Click to expand...

I haven't noticed any particular slowness, either today or earlier.

That said, some sort of tarpit that throttles abusive IPs is probably a useful website hardening step.
 
One thing I implemented on my web sites were honeypots that only robots could see (i.e. 1x1 pixel clear gifs somewhere on the page with a link to a page filled with e-mail addresses from notorious spammers). Go ahead and scrape those... A step further is sending them into the honeypot and then banning their IP's for a day.
 
This new bot doesn't seem to be attempting to explore every URL. Still, maybe a hidden link might catch it? I'll look into this if the problem continues.

My main goal is to detect extremely rapid page loading and temporarily ban that IP. I've seen a few modules that do this. The main trouble is our existing server just can't generate these forum pages rapidly enough, so I can't set a threshold high enough that no human would ever hit.

Soon we'll be on a much better server....
 
Last edited:
I've also noted some span in the threads. Someone's account got hacked and spam sent through it or they themselves are spammers.
 
The forum is been fully switched over to the new server.

If you notice any slowness or performance issues, please let me know?
 
Status
Not open for further replies.
Back
Top