GoFuckYourself.com - Adult Webmaster Forum - View Single Post - Offline Browsers spidering and crashing apache.

[Dan] · 02-15-2003, 10:55 PM

well it definately smells like something else. Since keepalive is off the spider is almost out of the question, cause for each file it will request, it will end up on a different apache child. So there is no way he will hung up a process like that and eat all the ressources (apache is better coded than that - if a bot was doing this it would mean everybody could crash apache servers easily). Not to mention that apache will never take 400mb even if it serves big files...

Like hel0 said, look for a php script with an infinite loop (really looks like that).. Or it could also be a rewrite rule that loops infinitely, or a 404 loop (i.e. a 404 generates a 404 which generates a 404 and so on). The php script looks like the strongest possibility. Maybe the bot triggers a script/URL that you usually don't use, that could explain why it happens only when the spider comes by.

02-15-2003, 10:55 PM
[Dan] Confirmed User Join Date: Oct 2001 Location: Somewhere in time Posts: 143	well it definately smells like something else. Since keepalive is off the spider is almost out of the question, cause for each file it will request, it will end up on a different apache child. So there is no way he will hung up a process like that and eat all the ressources (apache is better coded than that - if a bot was doing this it would mean everybody could crash apache servers easily). Not to mention that apache will never take 400mb even if it serves big files... Like hel0 said, look for a php script with an infinite loop (really looks like that).. Or it could also be a rewrite rule that loops infinitely, or a 404 loop (i.e. a 404 generates a 404 which generates a 404 and so on). The php script looks like the strongest possibility. Maybe the bot triggers a script/URL that you usually don't use, that could explain why it happens only when the spider comes by.