Web server CPU at 99% or out of memory
 

I?m running a web server with dual Xenon 2.8 CPUs, 4 Gigs of RAM, Red Hat 8.0, and Apache 1.3.x. Since installing the server, for the last 9 months, the server?s CPU has been 99% idle, and has not used more than 2 Gigs of physical memory. That?s with pushing out over 40 Gigs of content per day with 1.2 million page views.

Over the last few days I?ve been getting an out of memory error and the server grinds to a crawl. I changed the MaxClients settings on Apache from 1024 to 512 and that solved the memory problem. However, the CPU is now pegged at 99% user, 1% system, and 0% idle.

I?ve been struggling with this for about 3 days now. I talked to another webmaster running IIS and he says that his quad processor web server is also pegged. His problem started at about the same time as mine. Given that we?re on totally different platforms, I?m thinking that there is some type of DoS going on but all the log files look normal.

Is anyone experiencing this problem? Any ideas on what?s going on?

Thanks in advance for any assistance.


:helpme

Did you turn off keepalive in httpd.conf? I see a lot of high traffic servers that still have that turned on.

Ask your host to capture a sample of the traffic to/from your server. This is the only real way to confirm your server is being attacked, some types of DoS such as ICMP flooding may not normally be logged.

Depending on your OS and kernel config you may be able to do it yourself-

tcpdump -qenli rl0 > tcpdump.log

(replace 'rl0' with the interface name of your ether card)

Warning - if you're sending out 40Gb a day this will generate a HUGE log, so ^C it within a minute or two.

Understanding the log is a little more involved, it's not simple enough to explain here...

I'm not familiar with this "xenon" cpu you speak off

:BangBang: :BangBang:

Run the command "top" as root and see which process is taking up all your CPU cycles.

Once you find the process, look for the PID (Process ID) and if it isnt anything important then "kill -9 PID" to get rid of it.

My guess is a runaway apache process.

You may also want to move over to a real OS like FreeBSD!

Give me a message on ICQ:

4930562

There are a number of adjustments that you can do to make the server run faster. But you might be more interested in knowing why its currently slowing down.

We are very exerpienced at these types of things and run 3 out of the TOP 10 adult sites out there, so knowing how to squeeze the last breath of performance from a machine is our forte.

Give me a ring on ICQ and we'll do our best to find the eliminate the problem.

Believe you me when I say that your sentiment is shared.

Folks, thanks for all the help.

With the exception of sniffing the traffic, I've tried everything above. I need to spend time making money. I've switched to one of our backup server. I will pull the box offline for analysis.

Thanks again.

Typo, I meant Xeon.

When you did the top command was there one httpd process in particular that was consuming all of the CPU, or were all of the httpd's using a lot of CPU?

The number one cause of the symptoms you are describing is a looping ErrorDocument 404. Meaning a 404 page was pointing to a page that did not exist thus causing Apache to spin out of control.

As it should be, he is pushing 1200GB a month through a Xeon machine that is by no means a limp machine. When the day comes that you need to turn keep alives off on your server to just keep it online, that is the day you need to rethink how you are doing things.

Well that sounds interesting. :1orglaugh

It was only one httpd process in particular that was consuming all the CPU.

my first suggestion... get rid of asshat :winkwink:


this doesn't sound like your problem, but if you're running something like postnuke or phpnuke, or similar programs, it can do this as well. generally only on sites with decent to high traffic volumes.

i have my apache modded from the normal settings, i had problems from high traffic as well, here is my modifications.

KeepAlive On
KeepAliveTimeout 9
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 150
MaxRequestsPerChild 0

I'm almost at a close with this problem. It appears that Apache is under a DoS attack to the one site that is using all the resources. I recompiled Apache and included mod_dosevasive.

http://www.nuclearelephant.com/projects/dosevasive/

It seems to be running ok but it runs best with KeepAlive set to off. I need to tweak the configuration for mod_dosevasive. Looking at top, the resource usage is respectable but it's still somewhat high. An average of 69.4% idle, normally, it is 99.4% idle. Overall, that's better than 0.0% idle.

I thought brute force hacking was bad. I would welcome that compared to this.

I agree. That's really not much traffic for that configuration to begin with, depending on how much is static and how much is dynamic (PHP, etc.).

Eliminating KeepAlive isn't always the best solution, but the default timeouts - particularly on RedHat - are almost never fitting and degenerate to performance.

How do you handle brute force hacking? I'm using IProtect but I still have to set MaxClients to at least 384.

I get an average of 15 brute force hacking attempts each day.