Web server CPU at 99% or out of memory - GoFuckYourself.com

NemesisEnforcer · 12-02-2003, 12:03 AM

I?m running a web server with dual Xenon 2.8 CPUs, 4 Gigs of RAM, Red Hat 8.0, and Apache 1.3.x. Since installing the server, for the last 9 months, the server?s CPU has been 99% idle, and has not used more than 2 Gigs of physical memory. That?s with pushing out over 40 Gigs of content per day with 1.2 million page views.

Over the last few days I?ve been getting an out of memory error and the server grinds to a crawl. I changed the MaxClients settings on Apache from 1024 to 512 and that solved the memory problem. However, the CPU is now pegged at 99% user, 1% system, and 0% idle.

I?ve been struggling with this for about 3 days now. I talked to another webmaster running IIS and he says that his quad processor web server is also pegged. His problem started at about the same time as mine. Given that we?re on totally different platforms, I?m thinking that there is some type of DoS going on but all the log files look normal.

Is anyone experiencing this problem? Any ideas on what?s going on?

Thanks in advance for any assistance.

Babaganoosh · 12-02-2003, 12:14 AM

Did you turn off keepalive in httpd.conf? I see a lot of high traffic servers that still have that turned on.

rowan · 12-02-2003, 12:16 AM

Ask your host to capture a sample of the traffic to/from your server. This is the only real way to confirm your server is being attacked, some types of DoS such as ICMP flooding may not normally be logged.

Depending on your OS and kernel config you may be able to do it yourself-

tcpdump -qenli rl0 > tcpdump.log

(replace 'rl0' with the interface name of your ether card)

Warning - if you're sending out 40Gb a day this will generate a HUGE log, so ^C it within a minute or two.

Understanding the log is a little more involved, it's not simple enough to explain here...

senseidru · 12-02-2003, 04:13 AM

Quote:

Originally posted by NemesisEnforcer
dual Xenon 2.8 CPUs

I'm not familiar with this "xenon" cpu you speak off

notjoe · 12-02-2003, 04:38 AM

Quote:

Originally posted by NemesisEnforcer
I?m running a web server with dual Xenon 2.8 CPUs, 4 Gigs of RAM, Red Hat 8.0, and Apache 1.3.x. Since installing the server, for the last 9 months, the server?s CPU has been 99% idle, and has not used more than 2 Gigs of physical memory. That?s with pushing out over 40 Gigs of content per day with 1.2 million page views.

Over the last few days I?ve been getting an out of memory error and the server grinds to a crawl. I changed the MaxClients settings on Apache from 1024 to 512 and that solved the memory problem. However, the CPU is now pegged at 99% user, 1% system, and 0% idle.

I?ve been struggling with this for about 3 days now. I talked to another webmaster running IIS and he says that his quad processor web server is also pegged. His problem started at about the same time as mine. Given that we?re on totally different platforms, I?m thinking that there is some type of DoS going on but all the log files look normal.

Is anyone experiencing this problem? Any ideas on what?s going on?

Thanks in advance for any assistance.

Run the command "top" as root and see which process is taking up all your CPU cycles.

Once you find the process, look for the PID (Process ID) and if it isnt anything important then "kill -9 PID" to get rid of it.

My guess is a runaway apache process.

You may also want to move over to a real OS like FreeBSD!

sync · 12-02-2003, 05:26 AM

Give me a message on ICQ:

4930562

There are a number of adjustments that you can do to make the server run faster. But you might be more interested in knowing why its currently slowing down.

We are very exerpienced at these types of things and run 3 out of the TOP 10 adult sites out there, so knowing how to squeeze the last breath of performance from a machine is our forte.

Give me a ring on ICQ and we'll do our best to find the eliminate the problem.

Persecuted · 12-02-2003, 11:17 AM

Believe you me when I say that your sentiment is shared.

NemesisEnforcer · 12-02-2003, 05:33 PM

Folks, thanks for all the help.

With the exception of sniffing the traffic, I've tried everything above. I need to spend time making money. I've switched to one of our backup server. I will pull the box offline for analysis.

Thanks again.

NemesisEnforcer · 12-02-2003, 05:41 PM

Quote:

Originally posted by senseidru

I'm not familiar with this "xenon" cpu you speak off

Typo, I meant Xeon.

davidd · 12-02-2003, 05:43 PM

Quote:

Originally posted by NemesisEnforcer
Folks, thanks for all the help.

With the exception of sniffing the traffic, I've tried everything above. I need to spend time making money. I've switched to one of our backup server. I will pull the box offline for analysis.

Thanks again.

When you did the top command was there one httpd process in particular that was consuming all of the CPU, or were all of the httpd's using a lot of CPU?

The number one cause of the symptoms you are describing is a looping ErrorDocument 404. Meaning a 404 page was pointing to a page that did not exist thus causing Apache to spin out of control.

davidd · 12-02-2003, 05:49 PM

Quote:

Originally posted by Armed & Hammered
I see a lot of high traffic servers that still have that turned on.

As it should be, he is pushing 1200GB a month through a Xeon machine that is by no means a limp machine. When the day comes that you need to turn keep alives off on your server to just keep it online, that is the day you need to rethink how you are doing things.

stocktrader23 · 12-02-2003, 05:53 PM

Quote:

Originally posted by davidd

The number one cause of the symptoms you are describing is a looping ErrorDocument 404. Meaning a 404 page was pointing to a page that did not exist thus causing Apache to spin out of control.

Well that sounds interesting.

NemesisEnforcer · 12-06-2003, 04:02 PM

Quote:

Originally posted by davidd

When you did the top command was there one httpd process in particular that was consuming all of the CPU, or were all of the httpd's using a lot of CPU?

The number one cause of the symptoms you are describing is a looping ErrorDocument 404. Meaning a 404 page was pointing to a page that did not exist thus causing Apache to spin out of control.

It was only one httpd process in particular that was consuming all the CPU.

iroc409 · 12-06-2003, 04:06 PM

my first suggestion... get rid of asshat

this doesn't sound like your problem, but if you're running something like postnuke or phpnuke, or similar programs, it can do this as well. generally only on sites with decent to high traffic volumes.

fris · 12-06-2003, 04:06 PM

i have my apache modded from the normal settings, i had problems from high traffic as well, here is my modifications.

KeepAlive On
KeepAliveTimeout 9
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 150
MaxRequestsPerChild 0

NemesisEnforcer · 12-06-2003, 04:17 PM

I'm almost at a close with this problem. It appears that Apache is under a DoS attack to the one site that is using all the resources. I recompiled Apache and included mod_dosevasive.

http://www.nuclearelephant.com/projects/dosevasive/

It seems to be running ok but it runs best with KeepAlive set to off. I need to tweak the configuration for mod_dosevasive. Looking at top, the resource usage is respectable but it's still somewhat high. An average of 69.4% idle, normally, it is 99.4% idle. Overall, that's better than 0.0% idle.

I thought brute force hacking was bad. I would welcome that compared to this.

fletcher · 12-06-2003, 04:26 PM

Quote:

Originally posted by davidd

As it should be, he is pushing 1200GB a month through a Xeon machine that is by no means a limp machine. When the day comes that you need to turn keep alives off on your server to just keep it online, that is the day you need to rethink how you are doing things.

I agree. That's really not much traffic for that configuration to begin with, depending on how much is static and how much is dynamic (PHP, etc.).

Eliminating KeepAlive isn't always the best solution, but the default timeouts - particularly on RedHat - are almost never fitting and degenerate to performance.

NemesisEnforcer · 12-06-2003, 04:31 PM

Quote:

Originally posted by fris
i have my apache modded from the normal settings, i had problems from high traffic as well, here is my modifications.

KeepAlive On
KeepAliveTimeout 9
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 150
MaxRequestsPerChild 0

How do you handle brute force hacking? I'm using IProtect but I still have to set MaxClients to at least 384.

I get an average of 15 brute force hacking attempts each day.

12-02-2003, 12:03 AM	#1
NemesisEnforcer Confirmed User Industry Role: Join Date: Aug 2003 Location: Vegas and Los Angeles Posts: 2,122	Web server CPU at 99% or out of memory I?m running a web server with dual Xenon 2.8 CPUs, 4 Gigs of RAM, Red Hat 8.0, and Apache 1.3.x. Since installing the server, for the last 9 months, the server?s CPU has been 99% idle, and has not used more than 2 Gigs of physical memory. That?s with pushing out over 40 Gigs of content per day with 1.2 million page views. Over the last few days I?ve been getting an out of memory error and the server grinds to a crawl. I changed the MaxClients settings on Apache from 1024 to 512 and that solved the memory problem. However, the CPU is now pegged at 99% user, 1% system, and 0% idle. I?ve been struggling with this for about 3 days now. I talked to another webmaster running IIS and he says that his quad processor web server is also pegged. His problem started at about the same time as mine. Given that we?re on totally different platforms, I?m thinking that there is some type of DoS going on but all the log files look normal. Is anyone experiencing this problem? Any ideas on what?s going on? Thanks in advance for any assistance.

12-02-2003, 12:14 AM	#2
Babaganoosh ♥♥♥ Likes Hugs ♥♥♥ Industry Role: Join Date: Nov 2001 Location: /home Posts: 15,841	Did you turn off keepalive in httpd.conf? I see a lot of high traffic servers that still have that turned on. __________________ I like pie.

12-06-2003, 04:06 PM	#14
iroc409 Confirmed User Join Date: Jan 2003 Location: midwest side, yo Posts: 4,728	my first suggestion... get rid of asshat this doesn't sound like your problem, but if you're running something like postnuke or phpnuke, or similar programs, it can do this as well. generally only on sites with decent to high traffic volumes. __________________ <a href="http://www.iroc409.com/"><img src="http://www.iroc409.com/adv/120x60.gif" border=0></a> icq: 1 7 6 4 2 0 9 6 0 Gallery templates for ONLY $25! w00t!

12-06-2003, 04:06 PM	#15
fris Too lazy to set a custom title Industry Role: Join Date: Aug 2002 Posts: 55,372	i have my apache modded from the normal settings, i had problems from high traffic as well, here is my modifications. KeepAlive On KeepAliveTimeout 9 MinSpareServers 10 MaxSpareServers 20 StartServers 20 MaxClients 150 MaxRequestsPerChild 0 __________________ Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence. WP Stuff

12-02-2003, 12:16 AM	#3
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Ask your host to capture a sample of the traffic to/from your server. This is the only real way to confirm your server is being attacked, some types of DoS such as ICMP flooding may not normally be logged. Depending on your OS and kernel config you may be able to do it yourself- tcpdump -qenli rl0 > tcpdump.log (replace 'rl0' with the interface name of your ether card) Warning - if you're sending out 40Gb a day this will generate a HUGE log, so ^C it within a minute or two. Understanding the log is a little more involved, it's not simple enough to explain here...

12-02-2003, 05:26 AM	#6
sync Confirmed User Join Date: Aug 2001 Location: NY Posts: 204	Give me a message on ICQ: 4930562 There are a number of adjustments that you can do to make the server run faster. But you might be more interested in knowing why its currently slowing down. We are very exerpienced at these types of things and run 3 out of the TOP 10 adult sites out there, so knowing how to squeeze the last breath of performance from a machine is our forte. Give me a ring on ICQ and we'll do our best to find the eliminate the problem.

12-02-2003, 11:17 AM	#7
Persecuted So Fucking Banned Join Date: Dec 2003 Posts: 117	Believe you me when I say that your sentiment is shared.

12-02-2003, 05:33 PM	#8
NemesisEnforcer Confirmed User Industry Role: Join Date: Aug 2003 Location: Vegas and Los Angeles Posts: 2,122	Folks, thanks for all the help. With the exception of sniffing the traffic, I've tried everything above. I need to spend time making money. I've switched to one of our backup server. I will pull the box offline for analysis. Thanks again.

12-06-2003, 04:17 PM	#16
NemesisEnforcer Confirmed User Industry Role: Join Date: Aug 2003 Location: Vegas and Los Angeles Posts: 2,122	I'm almost at a close with this problem. It appears that Apache is under a DoS attack to the one site that is using all the resources. I recompiled Apache and included mod_dosevasive. http://www.nuclearelephant.com/projects/dosevasive/ It seems to be running ok but it runs best with KeepAlive set to off. I need to tweak the configuration for mod_dosevasive. Looking at top, the resource usage is respectable but it's still somewhat high. An average of 69.4% idle, normally, it is 99.4% idle. Overall, that's better than 0.0% idle. I thought brute force hacking was bad. I would welcome that compared to this.