Web server CPU at 99% or out of memory - GoFuckYourself.com - Adult Webmaster Forum

X

NemesisEnforcer

Confirmed User

Join Date: Aug 2003

Posts: 2122
Share

Post
#1

Web server CPU at 99% or out of memory

12-01-2003, 10:03 PM

I?m running a web server with dual Xenon 2.8 CPUs, 4 Gigs of RAM, Red Hat 8.0, and Apache 1.3.x. Since installing the server, for the last 9 months, the server?s CPU has been 99% idle, and has not used more than 2 Gigs of physical memory. That?s with pushing out over 40 Gigs of content per day with 1.2 million page views.

Over the last few days I?ve been getting an out of memory error and the server grinds to a crawl. I changed the MaxClients settings on Apache from 1024 to 512 and that solved the memory problem. However, the CPU is now pegged at 99% user, 1% system, and 0% idle.

I?ve been struggling with this for about 3 days now. I talked to another webmaster running IIS and he says that his quad processor web server is also pegged. His problem started at about the same time as mine. Given that we?re on totally different platforms, I?m thinking that there is some type of DoS going on but all the log files look normal.

Is anyone experiencing this problem? Any ideas on what?s going on?

Thanks in advance for any assistance.

The Only Time When Success Comes Before Work Is In A Dictionary.

Did you ever notice: When you put the 2 words 'The' and 'IRS' together it spells 'Theirs.'
Tags: None
Babaganoosh

♥♥♥ Likes Hugs ♥♥♥

Join Date: Nov 2001

Posts: 15841
Share

Post
#2

12-01-2003, 10:14 PM

Did you turn off keepalive in httpd.conf? I see a lot of high traffic servers that still have that turned on.

I like pie.
Comment
rowan

Too lazy to set a custom title

Join Date: Mar 2002

Posts: 17393
Share

Post
#3

12-01-2003, 10:16 PM

Ask your host to capture a sample of the traffic to/from your server. This is the only real way to confirm your server is being attacked, some types of DoS such as ICMP flooding may not normally be logged.

Depending on your OS and kernel config you may be able to do it yourself-

tcpdump -qenli rl0 > tcpdump.log

(replace 'rl0' with the interface name of your ether card)

Warning - if you're sending out 40Gb a day this will generate a HUGE log, so ^C it within a minute or two.

Understanding the log is a little more involved, it's not simple enough to explain here...
Comment
senseidru

Confirmed User

Join Date: Sep 2002

Posts: 1767
Share

Post
#4

12-02-2003, 02:13 AM

Originally posted by NemesisEnforcer
dual Xenon 2.8 CPUs

I'm not familiar with this "xenon" cpu you speak off
Comment
notjoe

Confirmed User

Join Date: May 2002

Posts: 5599
Share

Post
#5

12-02-2003, 02:38 AM

Originally posted by NemesisEnforcer
I?m running a web server with dual Xenon 2.8 CPUs, 4 Gigs of RAM, Red Hat 8.0, and Apache 1.3.x. Since installing the server, for the last 9 months, the server?s CPU has been 99% idle, and has not used more than 2 Gigs of physical memory. That?s with pushing out over 40 Gigs of content per day with 1.2 million page views.

Over the last few days I?ve been getting an out of memory error and the server grinds to a crawl. I changed the MaxClients settings on Apache from 1024 to 512 and that solved the memory problem. However, the CPU is now pegged at 99% user, 1% system, and 0% idle.

I?ve been struggling with this for about 3 days now. I talked to another webmaster running IIS and he says that his quad processor web server is also pegged. His problem started at about the same time as mine. Given that we?re on totally different platforms, I?m thinking that there is some type of DoS going on but all the log files look normal.

Is anyone experiencing this problem? Any ideas on what?s going on?

Thanks in advance for any assistance.

Run the command "top" as root and see which process is taking up all your CPU cycles.

Once you find the process, look for the PID (Process ID) and if it isnt anything important then "kill -9 PID" to get rid of it.

My guess is a runaway apache process.

You may also want to move over to a real OS like FreeBSD!
Comment
sync

Confirmed User

Join Date: Aug 2001

Posts: 204
Share

Post
#6

12-02-2003, 03:26 AM

Give me a message on ICQ:

4930562

There are a number of adjustments that you can do to make the server run faster. But you might be more interested in knowing why its currently slowing down.

We are very exerpienced at these types of things and run 3 out of the TOP 10 adult sites out there, so knowing how to squeeze the last breath of performance from a machine is our forte.

Give me a ring on ICQ and we'll do our best to find the eliminate the problem.

ICQ: 4930562
Got questions?? Sync got Answers
Comment
Persecuted

So Fucking Banned

Join Date: Dec 2003

Posts: 117
Share

Post
#7

12-02-2003, 09:17 AM

Believe you me when I say that your sentiment is shared.
Comment
NemesisEnforcer

Confirmed User

Join Date: Aug 2003

Posts: 2122
Share

Post
#8

12-02-2003, 03:33 PM

Folks, thanks for all the help.

With the exception of sniffing the traffic, I've tried everything above. I need to spend time making money. I've switched to one of our backup server. I will pull the box offline for analysis.

Thanks again.

The Only Time When Success Comes Before Work Is In A Dictionary.

Did you ever notice: When you put the 2 words 'The' and 'IRS' together it spells 'Theirs.'
Comment
NemesisEnforcer

Confirmed User

Join Date: Aug 2003

Posts: 2122
Share

Post
#9

12-02-2003, 03:41 PM

Originally posted by senseidru

I'm not familiar with this "xenon" cpu you speak off

Typo, I meant Xeon.

The Only Time When Success Comes Before Work Is In A Dictionary.

Did you ever notice: When you put the 2 words 'The' and 'IRS' together it spells 'Theirs.'
Comment
davidd

Confirmed User

Join Date: Jul 2003

Posts: 1076
Share

Post
#10

12-02-2003, 03:43 PM

Originally posted by NemesisEnforcer
Folks, thanks for all the help.

With the exception of sniffing the traffic, I've tried everything above. I need to spend time making money. I've switched to one of our backup server. I will pull the box offline for analysis.

Thanks again.

When you did the top command was there one httpd process in particular that was consuming all of the CPU, or were all of the httpd's using a lot of CPU?

The number one cause of the symptoms you are describing is a looping ErrorDocument 404. Meaning a 404 page was pointing to a page that did not exist thus causing Apache to spin out of control.
Comment
davidd

Confirmed User

Join Date: Jul 2003

Posts: 1076
Share

Post
#11

12-02-2003, 03:49 PM

Originally posted by Armed & Hammered
I see a lot of high traffic servers that still have that turned on.

As it should be, he is pushing 1200GB a month through a Xeon machine that is by no means a limp machine. When the day comes that you need to turn keep alives off on your server to just keep it online, that is the day you need to rethink how you are doing things.
Comment
stocktrader23

Let's do some business.

Join Date: Jan 2003

Posts: 18781
Share

Post
#12

12-02-2003, 03:53 PM

Originally posted by davidd

The number one cause of the symptoms you are describing is a looping ErrorDocument 404. Meaning a 404 page was pointing to a page that did not exist thus causing Apache to spin out of control.

Well that sounds interesting.

Hands Free Adult - Join Once, Earn For Life

"I try to make a habit of bouncing my eyes up to the face of a beautiful woman, and often repeat “not mine” in my head or even verbally. She’s not mine. God has her set aside. She’s not mine. She’s His little girl, and she needs me to fight for her by keeping my eyes where they should be."
Comment
NemesisEnforcer

Confirmed User

Join Date: Aug 2003

Posts: 2122
Share

Post
#13

12-06-2003, 02:02 PM

Originally posted by davidd

When you did the top command was there one httpd process in particular that was consuming all of the CPU, or were all of the httpd's using a lot of CPU?

The number one cause of the symptoms you are describing is a looping ErrorDocument 404. Meaning a 404 page was pointing to a page that did not exist thus causing Apache to spin out of control.

It was only one httpd process in particular that was consuming all the CPU.

The Only Time When Success Comes Before Work Is In A Dictionary.

Did you ever notice: When you put the 2 words 'The' and 'IRS' together it spells 'Theirs.'
Comment
iroc409

Confirmed User

Join Date: Jan 2003

Posts: 4728
Share

Post
#14

12-06-2003, 02:06 PM

my first suggestion... get rid of asshat

this doesn't sound like your problem, but if you're running something like postnuke or phpnuke, or similar programs, it can do this as well. generally only on sites with decent to high traffic volumes.

<a href="http://www.iroc409.com/"><img src="http://www.iroc409.com/adv/120x60.gif" border=0></a>

icq: 1 7 6 4 2 0 9 6 0
Gallery templates for ONLY $25! w00t!
Comment
fris

I have to go potty

Join Date: Aug 2002

Posts: 55818
Share

Post
#15

12-06-2003, 02:06 PM

i have my apache modded from the normal settings, i had problems from high traffic as well, here is my modifications.

KeepAlive On
KeepAliveTimeout 9
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 150
MaxRequestsPerChild 0

Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence.

My Newest Theme
Comment
NemesisEnforcer

Confirmed User

Join Date: Aug 2003

Posts: 2122
Share

Post
#16

12-06-2003, 02:17 PM

I'm almost at a close with this problem. It appears that Apache is under a DoS attack to the one site that is using all the resources. I recompiled Apache and included mod_dosevasive.

http://www.nuclearelephant.com/projects/dosevasive/

It seems to be running ok but it runs best with KeepAlive set to off. I need to tweak the configuration for mod_dosevasive. Looking at top, the resource usage is respectable but it's still somewhat high. An average of 69.4% idle, normally, it is 99.4% idle. Overall, that's better than 0.0% idle.

I thought brute force hacking was bad. I would welcome that compared to this.

The Only Time When Success Comes Before Work Is In A Dictionary.

Did you ever notice: When you put the 2 words 'The' and 'IRS' together it spells 'Theirs.'
Comment
fletcher

Confirmed User

Join Date: Jan 2003

Posts: 698
Share

Post
#17

12-06-2003, 02:26 PM

Originally posted by davidd

As it should be, he is pushing 1200GB a month through a Xeon machine that is by no means a limp machine. When the day comes that you need to turn keep alives off on your server to just keep it online, that is the day you need to rethink how you are doing things.

I agree. That's really not much traffic for that configuration to begin with, depending on how much is static and how much is dynamic (PHP, etc.).

Eliminating KeepAlive isn't always the best solution, but the default timeouts - particularly on RedHat - are almost never fitting and degenerate to performance.

 
[email protected]
ICQ: 6411138
Comment
NemesisEnforcer

Confirmed User

Join Date: Aug 2003

Posts: 2122
Share

Post
#18

12-06-2003, 02:31 PM

Originally posted by fris
i have my apache modded from the normal settings, i had problems from high traffic as well, here is my modifications.

KeepAlive On
KeepAliveTimeout 9
MinSpareServers 10
MaxSpareServers 20
StartServers 20
MaxClients 150
MaxRequestsPerChild 0

How do you handle brute force hacking? I'm using IProtect but I still have to set MaxClients to at least 384.

I get an average of 15 brute force hacking attempts each day.

The Only Time When Success Comes Before Work Is In A Dictionary.

Did you ever notice: When you put the 2 words 'The' and 'IRS' together it spells 'Theirs.'
Comment

Previous template Next

Working...