First, thanks for all the input regarding this. I don't know if it'll happen again, since the ip has been blocked now. I've tried to duplicate it myself and succeeded to get the server load up to 30+ with 100+ concurrent processes spidering without apache or mysql crashing.
Quote:
Originally posted by XXXManager
Hi. I was called here by Mr Fiction  so here I am...
> 7156 www 18 0 446M 371M 56 R 0 63.6 42.2 12:41 apache:
OUCH. 446M 371M? Goddamn.
MaxRequestsPerChild has a crucial effect mainly if you use resident modules like mod_perl on a persistent connection base. if not you can set it to 1 or 10 like I suggested - for a while to see the effect.
Actually I recommend turning off the keep-alive (state "KeepAlive Off" in httpd.conf) if you dont have graphics rich site. Again - you can do it for testing..
Restarting Apache every 15mins: Not nice.
Question:
1. what is your MaxClients in httpd.conf?
2. assuming you still use keep-alive - what is your KeepAliveTimeout?
3. what is your MaxSpareServers?
Since what you identified as "cause" of the crash can only be an irrelevant syptom - I suggest debugging the server..
Suggestion:
(If you want to fix this for long term)
1. log the top into a file (top -b > /path/to/log/top.log)
2. log the memory usage and swap activity every 10 seconds(vmstat -n 10 > /path/to/log/vmstats.log)
3. Enable the apache logs so you can see the last requests before the hang (I assume you already did that by the info you gave)
If anyone want/need to know anything I mentioned, ask.
|
Not using mod_perl, very little cgi at all actually.
.conf settings:
Timeout 90
KeepAlive off
KeepAliveTimeout 5
MinSpareServers 8
MaxSpareServers 16
StartServers 25
MaxClients 400
MaxRequestsPerChild 600
/server-status reported the same ip # / crawler each time connected to the processes that wouldn't die. Each request for the same page, with various different query strings appended that didn't exist. No mysql was involved in that connection.