Recent "An error occurred while the page was being generated" issue

Paul

Paul

Administrator
Staff member
Starting yesterday (May 27, 2025) the forum began having problems, showing "An error occurred while the page was being generated".

After some investigation, the problem turned out to be a "xf_session_activity" table grew too large. Apparently it's a common problem on some other Xenforo-based forums.

For a quick fix, I changed the mysql config files to allow up to 64MB in memory tables. Apparently the default was 16MB. Hopefully this will be enough to keep the problem from happening again.

Another suggestion I didn't do (yet) was a script that checks for the problem and truncates the xf_session_activity table.

Mentioned on 1 of those other Xenforo threads linked from msg #2 was some discussion about the likely root cause... spiders or spammers hitting the server really hard with lots of page requests, which apparently can fill up that memory-based session table. If this problem keeps coming back even with the table allowed to grow 4X larger, then I'll start to look at ways we might detect and throttle or block badly behaving bots.

At least right now we've back up and running, with only the max table size setting increased. I'm not eager to pour a lot more work into this problem until it's really more of a problem. But I did want to take some time to explain what went wrong (also for future me when/if more work is needed) since the on-screen "An error occurred..." message gave basically no useful info about the actual problem.
 
Looking at the apache log files, the access patterns look similar to the problems we had back in October 2024. Whoever is doing this commands an impressive army of servers with thousands of different IP numbers. IPdb says finds locations all over the world, Singapore, Vietnam, Brazil seem to be most of the ones attacking right now.

Looks like this recent attack began 4 or 5 days ago, but was at a lower level until yesterday. Normally the server logs about 220K lines per day. Starting May 24 the daily log files have about 450K lines. Looks like the server can handle that pace pretty well. But yesterday's log file has 625K, and today already has 603K lines.

I turned on the rather simple & crude fail2ban filters I created back in October. It does seem to slow this down a bit, but the accesses to the server still keep coming from new IP numbers. :(
 
The bots are still at it, fetching about 10-15 URLs per second from a vast pool of IP numbers.

I'm configured the server to reply with 403 error for 22 large netblocks which are more than half the requests so far. A quick 403 response means PHP doesn't run and no mysql queries, which really relieves the load from the server.
 
Would a slow 403 response be possible / more effective? Any bot waiting for a response isn't making another request ... or maybe I'm just parading my ignorance here :unsure:
 
Years ago I looked at slow responses for misbehaving bots. Maybe there is some way to do it, but it's tricky to accomplish without tying up resources on our side while the connection remains open.

For an idea of the scale, yesterday's log file shows we answered 263,619 http requests with 403 Forbidden over a 24 hour period. Those requests came from 101,294 unique IP numbers!

Many more were answered with 200 or 301, but from visually looking through a few small parts of the log file, I'm pretty sure we're answering more than 50% of this botnet's queries with 403. Difficult to know for sure, since this bot sends user agent into to mimic common browsers. It doesn't identify itself like the legitimate bots from Google, Microsoft Bing, Facebook, etc.

Fortunately the load right now is quite manageable. But if whoever is running this botnet ramps up the request rate, the forum probably will go down again. If we have more outages, I'll look into stronger anti-bot measures.
 
Last edited:
PaulStoffregen said:
Fortunately the load right now is quite manageable.
Click to expand...
The p#3 note was when it was horrid - not seen that since. That may have been after initial expanded memory alloc and before the 403 responses.

Was totally not good then (in cycles) if it did draw like clicking the @KurtE page it would draw only one post and none of the rest of the page, or just do "Page could not be loaded".
 
Hi @PaulStoffregen

Forum is completely inaccessible for me (in Brazil) for many days now. I tried to access it at home, at work, smartphone, and nothing works (different IPs, machines and browsers) . For a moment I was thinking that PJRC was gone forever 😭, until I tried to use a VPN.

Right now my VPN server is set to the Netherlands and I can access the forum normally.

Maybe the ip ban list is too broad right now?
Can't access the forum since last month.

Thanks
 
VictorFS said:
Hi @PaulStoffregen

Forum is completely inaccessible for me (in Brazil) for many days now. I tried to access it at home, at work, smartphone, and nothing works (different IPs, machines and browsers) . For a moment I was thinking that PJRC was gone forever 😭, until I tried to use a VPN.

Right now my VPN server is set to the Netherlands and I can access the forum normally.

Maybe the ip ban list is too broad right now?
Can't access the forum since last month.

Thanks
Click to expand...
Same here from Argentina, have to use a VPN to access the forum
 
I've disabled a couple of the large netblock filters. Hopefully this helps?

We're still receiving a huge amount of botnet traffic, but at least it's within the server's capability to keep up. I'm sad to say, if it becomes worse again like we saw 2 weeks ago, effectively taking the forum down for everyone, blocking huge netblocks seems to be the only viable defense.
 
Paul said:
I've disabled a couple of the large netblock filters. Hopefully this helps?

We're still receiving a huge amount of botnet traffic, but at least it's within the server's capability to keep up. I'm sad to say, if it becomes worse again like we saw 2 weeks ago, effectively taking the forum down for everyone, blocking huge netblocks seems to be the only viable defense.
Click to expand...
Now I can access normally, thanks
 
You must log in or register to reply here.
Back
Top