Using the top tool - my server overloading

[camperjohn64]

My server is overloading with very few users. When running #top, what do you guys see on your system? Can you post an example of what you see for comparison?

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5291 nobody    33  18 41236  15m 5796 S  3.3  0.8   0:21.91 httpd
27520 named     24   0 38108 3224 2192 S  0.3  0.2  22:06.92 named
 7670 nobody    33  18 41268  15m 5632 S  0.3  0.8   0:12.91 httpd
 8787 nobody    33  18 41112  15m 5592 S  0.3  0.8   0:05.54 httpd
 9518 nobody    33  18 41112  15m 5508 S  0.3  0.8   0:04.47 httpd
 9604 nobody    33  18 41112  15m 5444 S  0.3  0.8   0:05.17 httpd
 9808 nobody    33  18 41236  15m 5512 S  0.3  0.8   0:03.39 httpd
10328 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:03.12 httpd
10336 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:01.94 httpd

Do you see similar values on your servers?
Do you see the similar values for the VIRT column as I have above (for both NAMED and HTTPD)?
Does a typical httpd request take 0.2% memory on your system as well?

I'm hosting single static pages. No PHP.

Yes yes I know this is dependant on server config I am just looking for examples of what is within normal ranges.

[SmellyNose]

My server is overloading with very few users. When running #top, what do you guys see on your system? Can you post an example of what you see for comparison?

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5291 nobody    33  18 41236  15m 5796 S  3.3  0.8   0:21.91 httpd
27520 named     24   0 38108 3224 2192 S  0.3  0.2  22:06.92 named
 7670 nobody    33  18 41268  15m 5632 S  0.3  0.8   0:12.91 httpd
 8787 nobody    33  18 41112  15m 5592 S  0.3  0.8   0:05.54 httpd
 9518 nobody    33  18 41112  15m 5508 S  0.3  0.8   0:04.47 httpd
 9604 nobody    33  18 41112  15m 5444 S  0.3  0.8   0:05.17 httpd
 9808 nobody    33  18 41236  15m 5512 S  0.3  0.8   0:03.39 httpd
10328 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:03.12 httpd
10336 nobody    33  18 41112  15m 5432 S  0.3  0.8   0:01.94 httpd

Do you see similar values on your servers?
Do you see the similar values for the VIRT column as I have above (for both NAMED and HTTPD)?
Does a typical httpd request take 0.2% memory on your system as well?

I'm hosting single static pages. No PHP.

Yes yes I know this is dependant on server config I am just looking for examples of what is within normal ranges.

That looks fine.

What makes you say your server is overloaded?
What does "free -m" show?
What is your load average? (uptime or top)
Are you using swap? - This is shown in top aswell.

Also, I always run "top -c" rather than "top". It's much more helpful.

[Lace]

On a side note, the site in your sig has been suspended.

[liquidmoe]

Sometimes when you have high CPU processing on httpd processes and nothing else it could be indicative of poorly structured rewrite rules in one of your .htaccess files.

You may want to take a look at that.

[camperjohn64]

About every 12 hours, or 18 or whenever, my server gets "stuck". uptime goes to load average: 150.26, 150.37, 151.32 and everything stops. I do a httpd stop, wait, then httpd start and it runs fine:load average goes back to 0.01 etc.....but only for another X hours then it sticks again. Completely random. Could be in 20 minutes, or 18 hours.

I have been looking for it for about a month, and can't find what the deal is. I have shut down Mysql, suspended (almost) all accounts except for the account that gets most visitors, and it still gets stuck. I have a SINGLE static page, and bam, still crashes. No php, no nothing.

I say "stuck" because it is still running, and I can shell to it, just that httpd is no longer processing requests, and doesnt flush out the old ones. It's stuck.

I have hired 2 admins to help out, but both look at the server and are stumped after about an hour...

[camperjohn64]

On a side note, the site in your sig has been suspended.

Yes I did that, to rule out that website being the cause...most other sites are also suspended until I find this.

[Klen]

Is that dedicated or shared hosting?

[StaceyJo]

So technical for me here. hehh

[CYF]

that top output looks fine

Every 12 or 18 hours? You have anything setup in cron that is processor intensive?

Have you checked logs for bots? Sometimes they hammer a site and bring it to a crawl.

Are you running apache? What version? What OS?

[camperjohn64]

Sometimes when you have high CPU processing on httpd processes and nothing else it could be indicative of poorly structured rewrite rules in one of your .htaccess files.

You may want to take a look at that.

Good idea. The main site does use a 404 rewrite, and it's been working for 5 years now with up to 100,000 visitors a day at it's peak, but maybe I changed something in one of the other smaller sites.

that top output looks fine

Every 12 or 18 hours? You have anything setup in cron that is processor intensive?

Have you checked logs for bots? Sometimes they hammer a site and bring it to a crawl.

Are you running apache? What version? What OS?

Running CentOS, apache, basic LAMP server...CentOS release 4.8 (Final)

I will shut down all my cron jobs to make sure. I forgot about that! But shutting down MySQL should shut down any cron jobs that use MySQL data. But good idea!

I don't think it's bots or traffic, because if it was just too much traffic, then after restarting apache, it would get stuck straight away again. It happens at odd times.

Here is some interesting notes when I managed to "catch it in the act" of getting stuck. When it is starting to get stuck, I notice uptime will start to report higher and higher values. During this time I can get the apache status. Once its stuck, then I can't. But I did manage to catch it in the act, and here is what uptime showed me:

0.12 0.08 0.09
5.12 0.12 0.08
30.2 5.12 0.12
90.0 30.2 5.12
150.1 150.2 149.0
(server is now toast)

In a very short time, it's as if it gets "stuck". Once it's stuck, I shutdown apache, restart it, then everthing runs fine for 13 more hours, then it happens again.

lynx --dump localhost/whm-server-status

So, tell me what you make of these apache status reports:

http://www.photofight.com/xx.txt (started to get stuck)
http://www.photofight.com/xy.txt (a little more stuck)
http://www.photofight.com/xz.txt (now it's really stuck)

Notice the jump halfway down, where the milliseconds for a response becomes 1921913212 ?!?!? Search for 9068 and you will see q2growth.com GET /Printing-Supplies/Printing-Supplies.html HTTP/1.1

Shutdown apache and restarted:
http://www.photofight.com/xq.txt (yay its not stuck)

If it was simply too much traffic, then restarting apache would cause it to overload immediatly. So it can't be just a matter of too much traffic. There is something fishy and I can't figure it out. I can't work on ANY more websites until I figure this out too, since I don't have any money coming in while its 'stuck' and it is top priority.

I love the word stuck. Its the best way to describe it.

Ideas? Notice how requests build up, but dont get out! They just stack on top of each other!

[borked]

why/how is your apache instance serving up both HTTP/1.0 and HTTP/1.1 ???

Everything on srv1.ryd0.com is 1.0 whereas all the rest are 1.1

--edit
nm, it's client based

[rowan]

Hmm... just thinking about possible issues, and this one jumped out... why do you have mysql on the server if all pages are static? Do you run a script that rebuilds pages?

[borked]

What's weird is all your apache requests are pretty much "sending reply"
Normally, you have a big mix of "Reading Request",. Closing Connection, Waiting for Connection, and just a few sending replies.

Yours are all sending reply.

Here's an output from one of my servers that is handling ~60 requests/sec to your 7 requests/sec.

Parent Server Generation: 0
Server uptime: 1 day 21 hours 51 minutes 46 seconds
Total accesses: 9393716 - Total Traffic: 5.1 GB
CPU Usage: u1865.79 s183.61 cu3.53 cs0 - 1.24% CPU load
56.9 requests/sec - 32.2 kB/second - 579 B/request
16 requests currently being processed, 25 idle workers

R__C__CC______C_W__WR____CC...CC....CC.C_.....C__.__.._._.......
......_.........................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
................................................................
....

You can see the # requests currently being processed on yours is gradually increasing over time, which means something is blocking apache, and like mentioned it could be a fucked up rewrite rule that is looping or something.

In any case all the sending replies is not normal

[camperjohn64]

Hmm... just thinking about possible issues, and this one jumped out... why do you have mysql on the server if all pages are static? Do you run a script that rebuilds pages?

I normally run programs that use Mysql PHP. I just shut it down to make sure it wasn't a MySQL hang up or something.

I put up the static page to rule out the PHP too. Basically I removed everything to get it stable, then I would add items and find out what is causing it. Still no luck as even with a single static page, it is still fucking up.

I am a good PHP programmer, but a lousy admin. :-(

[beta-tester]

It's interesting, but I do have the same issue as you described. The server load normally is totally ok, around .01, or so. But at some point of time during the day it also gets soo high, and the apache got stuck. Until I restart it, or let it do it by itself, the HTTP pretty much doesn't work at all.

I tried to look into it but couldn't find anything that would cause this. It's really annoying ;(

[camperjohn64]

It's interesting, but I do have the same issue as you described. The server load normally is totally ok, around .01, or so. But at some point of time during the day it also gets soo high, and the apache got stuck. Until I restart it, or let it do it by itself, the HTTP pretty much doesn't work at all.

I tried to look into it but couldn't find anything that would cause this. It's really annoying ;(

Are you using Cpanel? I am using Cpanel at the moment.

[camperjohn64]

Holy shit I see a pattern in my hosts down!!!

4:30am, 12:30pm 8:30pm. Give or take 1 minute!!!! Wow a clue in a clueless world!

That means a backup, a cron job. Backup or something.

I predict a crash at 8:30 tonight!

[beta-tester]

Are you using Cpanel? I am using Cpanel at the moment.

yes Cpanel with WHM.

Could be backup, but the thing is that Apache gets overloaded, not other process, that's the problem.

[Domain Broker]

Do you have a shitty host? Why aren't they looking into this?

[rowan]

Holy shit I see a pattern in my hosts down!!!

4:30am, 12:30pm 8:30pm. Give or take 1 minute!!!! Wow a clue in a clueless world!

That means a backup, a cron job. Backup or something.

I predict a crash at 8:30 tonight!

So get checkin' your crontab then...

Under FreeBSD when you make a snapshot of a drive that has a zillion files it can sometimes freeze the OS for a minute or two, you can type at the cmdline but as soon as it needs to access a HD it just waits... I know you're on a completely different OS, just saying that backups are not always 100% seamless.

[CYF]

Holy shit I see a pattern in my hosts down!!!

4:30am, 12:30pm 8:30pm. Give or take 1 minute!!!! Wow a clue in a clueless world!

That means a backup, a cron job. Backup or something.

I predict a crash at 8:30 tonight!

crontab -l

If you need help decoding when stuff is scheduled to run I can help.

[camperjohn64]

Do you have a shitty host? Why aren't they looking into this?

Its a dedicated server so they are hands free. Are there hosts that take care of dedicated admin problems? I will switch in a heartbeat...

[Ca$h-collector]

interesting infos

[borked]

for every username in
ls /var/spool/cron

run (as root)

crontab -u <username> -l

to see which crontab is the culprit.

[camperjohn64]

I tell you one thing though, I've learned ALOT about my server while trying to track this down. I have focused my skills on PHP and development, not so much admin, but I am actually getting confident in handing some admin functions now.

[ilbb]

I recommend set cron every minute and log apache server status so you can see which pages/scripts are requested in the peaks. This will help you determine which script is causing server overload

[camperjohn64]

I recommend set cron every minute and log apache server status so you can see which pages/scripts are requested in the peaks. This will help you determine which script is causing server overload

I did this 100 times already. It is not a specific script that is overloading it.

ps -aux gives me this

USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
nobody    6852  0.4  0.9 41988 18228 ?       RN   04:38   0:45 /usr/local/apache/bin/httpd -k start -DSSL
nobody    8893  0.4  0.8 41884 17608 ?       RN   05:17   0:37 /usr/local/apache/bin/httpd -k start -DSSL
nobody    9950  0.4  0.8 41756 17884 ?       RN   05:38   0:32 /usr/local/apache/bin/httpd -k start -DSSL
nobody   10153  0.4  0.9 44352 19716 ?       RN   05:42   0:30 /usr/local/apache/bin/httpd -k start -DSSL
nobody   10682  0.3  0.8 41756 17032 ?       RN   05:52   0:27 /usr/local/apache/bin/httpd -k start -DSSL
nobody   10900  0.4  0.8 41780 17336 ?       RN   05:57   0:27 /usr/local/apache/bin/httpd -k start -DSSL
nobody   11720  0.4  0.8 41640 16820 ?       RN   06:11   0:24 /usr/local/apache/bin/httpd -k start -DSSL
root     11891  1.2  1.7 38516 34880 ?       S    06:14   1:11 spamd child
nobody   12856  0.4  0.9 44232 19332 ?       RN   06:37   0:19 /usr/local/apache/bin/httpd -k start -DSSL
nobody   12973  0.4  0.8 41652 16812 ?       RN   06:40   0:17 /usr/local/apache/bin/httpd -k start -DSSL
nobody   12975  0.4  0.8 41640 16952 ?       RN   06:40   0:18 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13359  0.4  0.9 44220 19472 ?       RN   06:48   0:15 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13507  0.4  0.8 41640 16736 ?       RN   06:52   0:15 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13770  0.4  0.8 41580 17052 ?       RN   06:59   0:11 /usr/local/apache/bin/httpd -k start -DSSL
nobody   13903  0.4  0.8 41528 16676 ?       RN   07:01   0:12 /usr/local/apache/bin/httpd -k start -DSSL
nobody   14287  0.5  0.8 41528 17040 ?       RN   07:11   0:10 /usr/local/apache/bin/httpd -k start -DSSL
nobody   14638  0.5  0.8 41524 16712 ?       RN   07:22   0:08 /usr/local/apache/bin/httpd -k start -DSSL
nobody   14773  0.7  0.8 41516 16644 ?       RN   07:27   0:08 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15054  0.7  0.8 41528 16956 ?       RN   07:33   0:06 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15333  0.9  0.8 41568 16412 ?       RN   07:38   0:04 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15649  1.3  0.8 41528 16208 ?       RN   07:44   0:02 /usr/local/apache/bin/httpd -k start -DSSL
mailnull 15667  0.0  0.2  9596 4076 ?        S    07:44   0:00 /usr/sbin/exim -bd -q60m
mailnull 15669  0.0  0.0     0    0 ?        Z    07:44   0:00 [exim] <defunct>
nobody   15719  1.4  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15722  1.4  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15723  1.1  0.7 41364 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15724  1.0  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15725  1.0  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15726  1.0  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15728  1.1  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15730  1.1  0.7 41364 15368 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15731  1.3  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15732  1.1  0.7 41364 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15733  1.2  0.7 41364 15344 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15734  1.3  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15735  1.3  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15736  1.2  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15737  1.2  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15740  1.1  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15741  1.1  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15742  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15744  1.1  0.7 41364 15332 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15746  1.0  0.7 41364 15388 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15747  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15751  1.1  0.7 41364 15340 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15752  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15753  0.9  0.7 41372 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15754  1.1  0.7 41364 15340 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15755  0.9  0.7 41364 15364 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15756  1.1  0.7 41364 15344 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15757  0.9  0.7 41364 15384 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15758  1.1  0.7 41364 15332 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15759  0.9  0.7 41364 15372 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL
nobody   15760  1.1  0.7 41364 15336 ?       RN   07:45   0:01 /usr/local/apache/bin/httpd -k start -DSSL

Is it normal that an httpd process will take 0.7% memory, and 1.1% CPU time?

Server got stuck again this morning at exactly 4:41am. Still didn't find what cron job it is...

# crontab -l
* * * * * /root/loadcheck.sh >/dev/null 2>&1
11,26,41,56 * * * * /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1
2,58 * * * * /usr/local/bandmin/bandmin
0 0 * * * /usr/local/bandmin/ipaddrmap
55 1 * * * /usr/local/cpanel/whostmgr/docroot/cgi/cpaddons_report.pl --notify
*/5 * * * * /usr/local/cpanel/bin/dcpumon >/dev/null 2>&1
30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache
45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel/bin/optimizefs
51 0 * * * /scripts/upcp
0 1 * * * /scripts/cpbackup
0 2 * * * /scripts/mailman_chown_archives
35 * * * * /usr/bin/test -x /usr/local/cpanel/bin/tail-check && /usr/local/cpanel/bin/tail-check
0 6 * * * /scripts/exim_tidydb > /dev/null 2>&1

At least now I can predict when it will crash..4am, 12, 8pm. Now I just gotta find out why.

[magpan]

Its a dedicated server so they are hands free. Are there hosts that take care of dedicated admin problems? I will switch in a heartbeat...

NationalNet, they look after anything and everything with your managed, dedicated server(s). And, the service turnaround time is UNBELIEVABLE.

[SmellyNose]

30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache

Comment out this one. It runs at 30 minutes past the hour, every 4 hours

Or, run it yourself and see how long it takes to run and if it causes your server to halt.

[PastorSinAlot]

choopa would knock this out for u

[camperjohn64]

30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache

Comment out this one. It runs at 30 minutes past the hour, every 4 hours

Or, run it yourself and see how long it takes to run and if it causes your server to halt.

But the server is down every 8 hours...

I commented out both lines:
30 */4 * * * /usr/bin/test -x /scripts/update_db_cache && /scripts/update_db_cache
45 */8 * * * /usr/bin/test -x /usr/local/cpanel/bin/optimizefs && /usr/local/cpanel

And I will see what happens...

[camperjohn64]

Or it is possible that the first task, takes more than 4 hours to run, and therefore the second time it is run, halts the server until the first is done. Causing the halt every 8 hours (2 tasks while second is waiting for first to finish).

Hmmm...

[SmellyNose]

It's unlikely the first task took that long, but it is possible. I didn't check your times properly (I skimmed the thread, so sue me :P ), so it is more than likely the second task that runs every 8 hours which is causing issues.

Let us know how it goes

[camperjohn64]

Well that wasn't it. Server uptime is 150.0 150.0 150.0 at 11:17am...Both cron jobs were removed..

Back to square one. Having a server that doesn't work, with so few visitors sucks.

[beta-tester]

try checking MaxClients in Apache. I just figured out in some of the error logs that it says maxclients are riched, try increasing this value. I increased it, and now I'm monitoring to see what happens.

[drogers]

Well that wasn't it. Server uptime is 150.0 150.0 150.0 at 11:17am...Both cron jobs were removed..

Back to square one. Having a server that doesn't work, with so few visitors sucks.

It could be robots maxing Apache childs and causing load. You should check the logs for this.

[camperjohn64]

The problem seems to be that apache threads are not exiting. They stay in the top of the output buffer.

Here check it out. Here are some outputs as the server is slowly crashing...

http://www.photofight.com/xx.txt
http://www.photofight.com/xy.txt
http://www.photofight.com/xz.txt

See how the top thread is never leaving?

[SmokeyTheBear]

check if your getting botted

netstat -anp |grep ‘tcp\|udp’ | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n

[borked]

See how the top thread is never leaving?

that is what I said - the server is forever sending replies and taking forever to finish sending each reply.

Are you sure you don't have a rogue js somewhere? I know I can visit a site, and my browser will be forever "receiving item 19 of 20" or something similar. Means some js or remote DNS call is taking forever to complete. But it can bring down a server if that server is having enough hits at that time...

[HomerSimpson]

you need to tweak your apache...

[Varius]

What do you have set for the following values and which version of Apache?

MaxRequestsPerChild
KeepAlive
KeepAliveTimeout
MaxKeepAliveRequests

[camperjohn64]

MaxClients 250
MaxRequestsPerChild 100
KeepAlive On
KeepAliveTimeout 3
MaxKeepAliveRequests (not set)

[fatfoo]

Server overloading - that's not good. Good luck.

[Varius]

MaxClients 250
MaxRequestsPerChild 100
KeepAlive On
KeepAliveTimeout 3
MaxKeepAliveRequests (not set)

Looks ok but without being on the server, hard to say. Try turning KeepAlive Off and see if your problem gets solved...