PDA

View Full Version : PJRC Website Speed & Reliability, Feedback Wanted?



PaulStoffregen
10-08-2013, 07:54 PM
This morning the PJRC website was offline for several minutes. It appears some errant bot tried to rapidly access forum pages, which placed far too much load on the server. Ultimately, the server had to be rebooted by the hosting service.

I'd like to ask, have you notice slowness or non-responding pages?

We're looking into options to improve the site's hosting....

stevech
10-08-2013, 08:04 PM
none noted here.

nlecaude
10-08-2013, 08:30 PM
Been around all morning and didn't notice anything.

Constantin
10-08-2013, 08:54 PM
My first round of suggestions for the ADC improvements were wiped out. Had to do with the auto-logout I usually experience here with longer posts. But usually, the post is saved and then posted after I login again. Not this time. Oh Well.

Headroom
10-08-2013, 08:58 PM
Noted, but only very briefly.

MichaelMeissner
10-08-2013, 09:08 PM
Same as Headroom, I noticed it was down this morning and when I came back to it, it was up.

bigpilot
10-08-2013, 09:10 PM
I couldn't access the site for a while.

PaulStoffregen
10-08-2013, 09:15 PM
Was this morning just an isolated incident? Have you noticed slowdowns at other times over the last few months?

Constantin
10-08-2013, 09:42 PM
Isolated. I would not worry.

bigpilot
10-09-2013, 08:57 AM
I haven't noticed any other slowdowns or timeouts.

PaulStoffregen
10-09-2013, 12:04 PM
Looks like we had another slowdown this morning, around 3:45 am (pacific time), lasting about 4 minutes, and another close to 4am, also about 3-4 minutes.

From the server log file, it looks like an overly aggressive bot. The pattern seems to involve rapidly fetching several dozen random pages. It's playing tricks like using a different browser name in the user agent field on each request. The pages it accesses are a mix of random forum pages, plus URLs from the main site (which return 404 errors to the bot, because it's using "forum.pjrc.com" on URLs that should begin with "www.pjrc.com"). After several pages, it always goes to the forum login page and makes a few attempts. Then it repeats the process, hitting many random pages, then making more login attempts.

Damn spammers!

At least the good news is we're not getting any significant amount of spam on the forum anymore, so obviously they're not managing to get in. :)

daperl
10-09-2013, 02:41 PM
Not sure if it applys, but I just saw this:

http://it.slashdot.org/story/13/10/09/1253209/dangerous-vbulletin-exploit-in-the-wild

stevech
10-09-2013, 03:07 PM
I don't get it. What's the probability of getting money from a spam-ad here? 0.00000001?
Or is it just juvenile IT vandalism?

nox771
10-09-2013, 06:17 PM
Just FYI - was down again today at 11am CST.

PaulStoffregen
10-09-2013, 06:18 PM
Not sure if it applys, but I just saw this:
http://it.slashdot.org/story/13/10/09/1253209/dangerous-vbulletin-exploit-in-the-wild

This is unrelated. vBulletin actually sent a security advisory message about this weeks ago. I deleted the install directory that day.

They also sent an advisory last night about a XSS attack. But the timing was just coincidence. The vulnerable part is within an optional feature we never enabled.

Constantin
10-09-2013, 06:23 PM
The site was just down for a while or being DDOS'd

PaulStoffregen
10-09-2013, 06:42 PM
Just FYI - was down again today at 11am CST.


The site was just down for a while or being DDOS'd

Thanks. Looks like there were 3 separate bot attacks this morning around that time.

Sadly, it's looking like our little server is perfectly capable of handling normal forum usage where several real humans do things at normal human speeds. But as soon as 1 bot starts rapidly pounding the forum, the load is too great for the website to keep up. It is a dedicated server, but it's an old machine (single core Pentium 4) with only 1 GB of RAM. We'd never survive a DDOS.....

It's looking like we really need to buy a 2nd much faster server just for the forum. Our old server just isn't fast enough to respond rapidly enough (with the substantial overhead this forum takes) for an anti-bot detection to reliably tell the difference from a human who just happens to click several pages quickly.

I've been looking into hosting companies since last night. My hope is to get a server dedicated to the forum online and the forum moved over within the next few days.

stevech
10-09-2013, 08:12 PM
I'm sure you've thought about black-listing the offending remote host IP address within your router. Mine has such a black list. I have most of China in it.

For a year+ I've been using a Windows 7 Enterprise VM from Scale Matrix in San Diego. Month-to-month pricing - not the cheapest. They charge less for Linux. $100-200/mo with do-your-own backups. Much more if you want an SLA in the contract.
Amazon EC2 is low cost but hard to use.
Rackspace essentially resells Amazon S3 and EC2.
I'm going to change to use a provider that doesn't use the high cost, complex VMware approach.
A problem with all these kinds of VMs is that if/when you need to reboot the VM and boot into a different OS/program, such as if you have to boot a recovery standalone program... you have to pay them to do so via the VM admin which you can't access.

One that I like uses a KVM OVER IP, so when you reboot, you keep total control via KVM, and you're not dependent on the VM manager GUI being available to YOU, remotely. I mean a physical KVM so it matters not what OS or standalone program is running on the VM. Such as a DVD image file that has a program for disk recovery for when you lose the OS due to hacking or due to a failure of the provider's RAID (which happened once to me - my own backup saved my behind).

PaulStoffregen
10-09-2013, 08:19 PM
In case anyone's curious, the new forum server will be a E3-1230 Xeon with 16 GB RAM and a 80 GB SSD. That's a pretty big step up from the P4 with 1GB RAM and 7200 rpm drive we're on now. Hopefully the SSD will let mysql respond many more queries/sec and we can turn on PHP opcode caching and tune mysql better with the extra RAM.

I'm looking into bandwidth limiting options. But I don't want any bandwidth limiting to kick in for real humans, even if they click pretty fast. Anyone have any experience with such things?

Nantonos
10-10-2013, 01:17 AM
This morning the PJRC website was offline for several minutes. It appears some errant bot tried to rapidly access forum pages, which placed far too much load on the server. Ultimately, the server had to be rebooted by the hosting service.

I'd like to ask, have you notice slowness or non-responding pages?

We're looking into options to improve the site's hosting....

I haven't noticed any particular slowness, either today or earlier.

That said, some sort of tarpit that throttles abusive IPs is probably a useful website hardening step.

Constantin
10-10-2013, 02:09 AM
One thing I implemented on my web sites were honeypots that only robots could see (i.e. 1x1 pixel clear gifs somewhere on the page with a link to a page filled with e-mail addresses from notorious spammers). Go ahead and scrape those... A step further is sending them into the honeypot and then banning their IP's for a day.

bigpilot
10-10-2013, 06:51 AM
The site was inaccessible a couple of times yesterday.

PaulStoffregen
10-10-2013, 09:37 AM
This new bot doesn't seem to be attempting to explore every URL. Still, maybe a hidden link might catch it? I'll look into this if the problem continues.

My main goal is to detect extremely rapid page loading and temporarily ban that IP. I've seen a few modules that do this. The main trouble is our existing server just can't generate these forum pages rapidly enough, so I can't set a threshold high enough that no human would ever hit.

Soon we'll be on a much better server....

bigpilot
10-10-2013, 11:44 AM
I've also noted some span in the threads. Someone's account got hacked and spam sent through it or they themselves are spammers.

PaulStoffregen
10-10-2013, 07:02 PM
The forum is been fully switched over to the new server.

If you notice any slowness or performance issues, please let me know?

Constantin
10-10-2013, 07:16 PM
Seems to be fine. Any tarpits or honeypots for us to explore? :)

PaulStoffregen
10-10-2013, 08:23 PM
Nope, none. I have an incredibly long written list of small and large software developments planned for Teensy. Gaming the spammers just isn't anywhere on my list.

Ideally, some of these developments will help sell more Teensys. I'd much rather focus on that, and if necessary, use some of the money to simply upgrade the server to insanely overkill specs if this becomes a problem again. It's actually relatively cheap to get a pretty amazing dedicated server these days.

pictographer
10-10-2013, 09:10 PM
All looks good here.

christoph
10-12-2013, 08:45 AM
I also noticed the site not being accessible from time to time. I'll report here when I see this happening again.

nox771
10-21-2013, 02:08 PM
Are the forum notices working properly? I've not been getting notices for subscribed threads. I tried unsubscribing/resubscribing to one of the threads to see if it fixes it. I'm wondering if I should unsubscribe everything and re-add it.

Robin
10-21-2013, 02:11 PM
I'll take a look at the settings for forum notices and subscriptions.