server question
 

ok i'm wondering what is causing the high loads on my server and what i could do to fix it, and make the server faster.

i'm on a p4 2.6ghz with 2gb of ram.
(2) 80 GB IDE harddrives

i'm running around 200,000 hits a day to my movie galleries as an estimate which all have 4x1mb movies on them.

i also run 4 tgps which are mostly fairly small.
one gets around 70-80k/day and the others just get under 8k/day
the sites use tm3 and comus thumbs to run them.

this is the result from top. the load hovers between 5-15 usually and then spikes at 100 sometimes.

20:04:41 up 27 days, 3:17, 1 user, load average: 17.01, 11.12, 10.78
375 processes: 360 sleeping, 14 running, 1 zombie, 0 stopped
CPU states: 6.3% user 15.6% system 0.0% nice 0.0% iowait 77.9% idle
Mem: 2063328k av, 2038176k used, 25152k free, 0k shrd, 69028k buff
1534512k actv, 13936k in_d, 50144k in_c
Swap: 2040244k av, 33532k used, 2006712k free 1600892k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
20463 root 15 0 3960 240 188 S 3.3 0.0 1:13 0 httpd
6 root 15 0 0 0 0 SW 0.7 0.0 62:23 0 kscand/Normal
1785 root 15 0 1352 1352 800 R 0.7 0.0 0:03 0 top
3630 root 15 0 100 24 12 S 0.5 0.0 209:03 0 portsentry
1 root 15 0 108 80 56 S 0.0 0.0 0:08 0 init
2 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 keventd
3 root 34 19 0 0 0 SWN 0.0 0.0 0:13 0 ksoftirqd_CPU
8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
4 root 15 0 0 0 0 SW 0.0 0.0 3:41 0 kswapd
5 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand/DMA
7 root 16 0 0 0 0 SW 0.0 0.0 342:44 0 kscand/HighMe
9 root 15 0 0 0 0 SW 0.0 0.0 0:03 0 kupdated
10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
14 root 15 0 0 0 0 SW 0.0 0.0 13:59 0 kjournald
84 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
3002 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald




any help or advice would be great.
i'm at he.net on 100mbps server and it can't even manage to push 50mbps when its in peak time at loads of 50 fairly constant(from what i saw last night)

thanks
Paul

Sounds like you definitely didn't optimize your Apache binary nor your kernel

ICQ: 4930562

And your running linux lol -- get FreeBSD and problems b gone

are you running BSD or Linux? I can show you probably 50 or more ways to debug/optimize problems on a BSD based host.

Swap: 2040244k av, 33532k used, 2006712k free 1600892k cached

You're running into swapspace for one thing, swapspace is a lot slower than ram, you most likely want to upgrade your ram if you're learning on swap space that much!

you need more ram thats all i can tell you :)

yeah i'm running linux.

i've heard a couple people say about freebsd being better.

they said that i can't upgrade my ram any further, even though when i got the server they said i could get up to 4gb. i had a feeling it may be because its getting into the swap file.

thanks for the quick replies. i've been looking into this all day and getting nowhere fast

if your processes/network demand is more than your ram can deliver, FreeBSD still will not be able to solve the problem. While it is true FreeBSD does handle resources better (please not Linux vs. FreeBSD Jihad, this isn't slashdot! )
bottlenecks in hardware will surface in any operating system the same.

While in top type "M" .. that's capital M so "shift+M".

That will sort by memory usage. Find out what processes are consuming the most memory and then try to figure out why.

I'm also wondering why your stats are showing 77% cpu idle (which is not good) yet the top process doesn't seem to be using much cpu at all (it sorts by cpu usage by default).

I would be very interested in seeing your apache configuration .. since you only seem to have one apache process running.. or at least one near the top ..

I would have thought being 77% idle would point at the CPU not being the bottleneck, I have servers that have processors at 96% idle, and are running and managing their load just fine?

looks like a memory issue also note don't expect a machine to push 100mb/sec your hd's will probably crash and burn before that happends well atleast with standard ide drives.

some of you are fucking funny.

and have zero idea about sysadmin, sorry.


read up on load average and scheduling.

also, why in the fuck are you running apache as root? thats just plain dumb.

the memory question is a good one.
and, you'll never hit 100mb/s from a single 100mb/s nic. ever. there is protocol overhead there that will limit you down. how much really depends on the tcp stack.

this is an HE dedicated box right? you absolutely sure its dedicated? HE wraps their binaries to only show what you are running, not others.

ah, but apache as root. HE wouldn't do that.

uh huh. the guy runs apache as root and you want him dicking with kernel source?

toddler, your kinda wrong

the main apache process is ran by root (OK), the rest of the childs are forked off as nobody or www or whoever you set

also as far as pushing 100mbits/sec off 1 link sure, protocol overhead is still real traffic hence if you do 90mbps + 10mbps protocl your still pushing 100mbps on MRTG

and i've seen about 97mbps utilized for HTTP on MRTG :) (on a 100mbps link)

not sure what you're talking about, all apache configurations run one process as root, it is the process that passes the ability to open port 80 (privileged port) to the child, it never answers any queries, just passes on the privs to the child?

ok i sorted by memore usage.

21:39:05 up 27 days, 4:51, 1 user, load average: 11.27, 8.70, 8.81
313 processes: 307 sleeping, 4 running, 2 zombie, 0 stopped
CPU states: 2.1% user 7.5% system 0.0% nice 0.0% iowait 90.2% idle
Mem: 2063328k av, 2045652k used, 17676k free, 0k shrd, 69976k buff
1515572k actv, 48k in_d, 48632k in_c
Swap: 2040244k av, 24244k used, 2016000k free 1636312k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
19463 apache 15 0 9436 7308 4444 S 0.1 0.3 0:00 0 httpd
17940 apache 15 0 9204 7080 4136 S 0.0 0.3 0:00 0 httpd
19487 apache 15 0 8744 6240 3760 S 0.0 0.3 0:00 0 httpd
17929 apache 15 0 7184 4992 2544 S 0.0 0.2 0:00 0 httpd
17981 apache 15 0 7068 4876 2416 S 0.0 0.2 0:00 0 httpd
17912 apache 15 0 6972 4772 2800 S 0.0 0.2 0:00 0 httpd
19518 apache 15 0 7012 4772 2816 S 0.0 0.2 0:00 0 httpd
17975 apache 15 0 6800 4728 2268 S 0.0 0.2 0:00 0 httpd
17961 apache 15 0 6940 4696 2792 S 0.0 0.2 0:00 0 httpd
19515 apache 15 0 6920 4644 2784 S 0.1 0.2 0:00 0 httpd
17759 apache 15 0 6832 4636 2776 S 0.0 0.2 0:00 0 httpd
17771 apache 15 0 6812 4596 2116 S 0.0 0.2 0:01 0 httpd
18998 apache 15 0 6776 4588 2840 S 0.0 0.2 0:00 0 httpd
20237 apache 15 0 6840 4580 2912 S 0.1 0.2 0:00 0 httpd
19488 apache 15 0 6816 4568 2856 S 0.1 0.2 0:00 0 httpd
19517 apache 15 0 6800 4568 2860 S 0.0 0.2 0:00 0 httpd

1: yes, one. note load, note his touted traffic and ram usage. and no other httpd's in the first 20.

what exactly is a 100mb/s link? local or wan? :)


(xeno, yes you are correct. however something in the top does not add up. Not with a load that high. )

do me a favor and do a
ps -ef | grep -c apache and post
the number?

you might be able to save yourself a little memory from apache by unloading any DSO's you are not using.
Apache by default loads quite a few modules you might never use, look @ http://httpd.apache.org/docs/mod/

and read about each module, and determine if you need it, also try the following:
http://httpd.apache.org/docs/misc/perf-tuning.html
http://httpd.apache.org/docs/misc/perf.html#Linux

357

i didnt mention that i'm on redhat 9 with apache 2.0.40
php 4.3.4

2.x would be:
http://httpd.apache.org/docs-2.0/misc/perf-tuning.html
http://httpd.apache.org/docs-2.0/dso.html

Also see if you have iostat installed.

Do a iostat -x 5 and watch the read times and try to see if a particular partition is hitting high numbers.

If you don't have it installed it comes with the sysstat rpm. Would need more specifics like OS type, etc. I am a full time Linux SysAdmin so I could help you out.

Oops. Just read the OS type. Send me the top part of your httpd.conf without the virtual host entries. Also you might look at using PHP Accellerator or MMCACHE.

Apache's performance starts to go into the toilet after, if I recall correctly about 250+ concurrent processes..

You might want to look into switching over to Zeus for your webserver.

http://www.zeus.com/library/articles/zeusvsapache.html

Link might seemm a little biased, but I have seen a lot of talk about this before. On geek friendly forums. :D

iostat is a nice way to diagnose disk bottlenecks, I use it quite a bit on BSD, wasn't sure it was available as that command on Linux, nice call!

look at the iowait in his top. i don't think disk is really the issue.

You're kidding right, apache is the most popular web server on the internet according to surveys done by netcraft, a lot of big companies use it, zeus has a 1.60% share?

Hmmm. So hitting 4000 concurrent on a dual AMD machine is why I can only do 40mbps per machine? Apache can handle it's own if someone spends a little time researching on it.

not saying you're not right but, well...look at how popular windows is. Just because a ton of people use it doesn't make it any better. (though in this case I think it is, and apache shouldn't have a problem with that load. the child count seems high to me, but his traffic may or may not warrant it)

This is true. But alas IDE is just not prefered for a good server. SCSI all the way. Also posted was that he had 2 80 gig IDE drives. Are these mirrored? Are they on the same channel? A lot of things to know to solve high load problems.

Brad

true enough. Does HE offer scsi? I don't think this is 'his' machine, but a dedicated rental.

that being said, you'll notice in his top he as 313 procs, a couple responses down he had 357 apache processes. would be curious to see how many connections this box has.

Your best bet would be to hire someone to look at your machine for you, I think one of the fellows in this threads offered Linux Administration, and bought up some good points,
The fact is it really could be any one of hundreds of possible problems, from a bad NIC, non-duplexed NIC, bad memory, IDE channel drivers, and that's just hardware, not even looking at potential problems with the software itself.

Your best bet would be to spend a little cash on a professional to do a diagnosis on your machine, they most likely will find the problem, and perhaps even be able to tune it better.

hi Paul

what you can do, is to optimize your .htaccess files. that is causing the higher load for sure. expecially if you have high hits.

if you have no idea what im talking about, hit me on icq #132516819

bye

It is useless to discuss how to fix this with such limited information.

How do you plan to base your stuff on the top 20 or so lines from top?!

It is obvious that RAM is causing the problem, but the high load does not really mean a lot, CPU usage is very low.

Paul, is the box _SLOW_? Or are you just worried about the high load? Does it get slow at times?

sorry i had to head out for dinner




[root@ns1 root]# iostat -x 5
Linux 2.4.20-28.9 (ns1.bigxxxsite.com) 03/24/2004

avg-cpu: %user %nice %sys %idle
7.70 0.06 13.58 78.65

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.57 0.14 4.29 509387.48 0.45
/dev/hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 13.05 0.00 4.21 4.21 0.00
/dev/hda 36.88 71.58 5.85 24.79 341.90 773.86 170.95 386.93 36.41 0.11 0.46 0.03 0.09
/dev/hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 4.16 0.00 128.92 78.73 0.00
/dev/hda2 36.88 70.25 4.90 23.82 334.30 755.24 167.15 377.62 37.93 0.17 0.55 0.40 1.15
/dev/hda3 0.00 1.33 0.95 0.97 7.60 18.62 3.80 9.31 13.64 0.17 8.58 9.32 1.79

avg-cpu: %user %nice %sys %idle
0.60 0.00 6.00 93.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8589804.59 0.00 0.00 100.00
/dev/hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda 42.00 63.80 2.80 11.60 358.40 603.20 179.20 301.60 66.78 8589807.99 23.61 69.44 100.00
/dev/hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda2 42.00 63.20 2.80 11.00 358.40 593.60 179.20 296.80 68.99 3.28 23.77 10.14 14.00
/dev/hda3 0.00 0.60 0.00 0.60 0.00 9.60 0.00 4.80 16.00 0.12 20.00 20.00 1.20

avg-cpu: %user %nice %sys %idle
2.00 0.00 10.00 88.00

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8589804.59 0.00 0.00 100.00
/dev/hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda 54.40 18.80 3.80 2.80 465.60 172.80 232.80 86.40 96.73 8589805.85 19.09 151.52 100.00
/dev/hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda2 54.40 17.20 3.80 1.60 465.60 150.40 232.80 75.20 114.07 1.02 18.89 8.52 4.60
/dev/hda3 0.00 1.60 0.00 1.20 0.00 22.40 0.00 11.20 18.67 0.24 20.00 13.33 1.60

avg-cpu: %user %nice %sys %idle
2.00 0.00 7.60 90.40

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8589804.59 0.00 0.00 100.00
/dev/hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda 24.00 37.60 2.00 9.20 208.00 374.40 104.00 187.20 52.00 8589805.65 9.46 89.29 100.00
/dev/hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda2 24.00 37.00 2.00 8.00 208.00 360.00 104.00 180.00 56.80 0.82 8.20 3.40 3.40
/dev/hda3 0.00 0.60 0.00 1.20 0.00 14.40 0.00 7.20 12.00 0.24 20.00 16.67 2.00

avg-cpu: %user %nice %sys %idle
1.60 0.00 8.60 89.80

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8589804.59 0.00 0.00 100.00
/dev/hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda 12.00 40.80 1.40 6.40 107.20 377.60 53.60 188.80 62.15 8589805.43 10.77 128.21 100.00
/dev/hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda2 12.00 40.80 1.40 5.20 107.20 368.00 53.60 184.00 72.00 0.60 9.09 4.85 3.20
/dev/hda3 0.00 0.00 0.00 1.20 0.00 9.60 0.00 4.80 8.00 0.24 20.00 13.33 1.60

avg-cpu: %user %nice %sys %idle
3.60 0.00 12.20 84.20

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/hdc 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 8589804.59 0.00 0.00 100.00
/dev/hdc1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda 12.00 34.40 1.20 5.60 105.60 320.00 52.80 160.00 62.59 8589805.27 10.00 147.06 100.00
/dev/hda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
/dev/hda2 12.00 33.60 1.20 5.20 105.60 310.40 52.80 155.20 65.00 0.60 9.38 3.75 2.40
/dev/hda3 0.00 0.80 0.00 0.40 0.00 9.60 0.00 4.80 24.00 0.08 20.00 20.00 0.80

is the result of the iostat test

Timeout 100

KeepAlive Off

MaxKeepAliveRequests 150

KeepAliveTimeout 2


<IfModule prefork.c>
StartServers 20
ServerLimit 2500
MinSpareServers 20
MaxSpareServers 300
MaxClients 2500
MaxRequestsPerChild 300
</IfModule>

is the main settings in httpd.conf

no they aren't mirrored. the swap file is on the second hd

Run FBSD without swap.. that is what i do on a couple of machines... once a server hits the swap drive it's game over.

i paid someone before, and all they did was put a monitor on it which rebooted the server every few minutes.

yeah the box slows down at peak times, and then on and off it goes slow for small periods of time(ie 20 seconds or so) during non peak times too.

for trades etc, it really stuffs up site.

just hit a load of 112 then. don't know if i got the right top report here, to show anything. root is the owner of httpd when it is in the non-threaded view of top.

04:38:11 up 27 days, 11:50, 1 user, load average: 112.25, 60.50, 31.01
816 processes: 799 sleeping, 16 running, 1 zombie, 0 stopped
CPU states: 6.0% user 15.9% system 0.0% nice 0.0% iowait 77.9% idle
Mem: 2063328k av, 2053804k used, 9524k free, 0k shrd, 131364k buff
1297008k actv, 263188k in_d, 46592k in_c
Swap: 2040244k av, 23536k used, 2016708k free 1505172k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
20463 root 15 0 9780 9044 5816 S 4.5 0.4 0:29 0 httpd
11993 root 15 0 1780 1780 796 R 1.7 0.0 0:04 0 top
13135 root 34 19 720 720 432 R N 1.1 0.0 0:02 0 updatedb
3630 root 15 0 100 24 12 R 0.9 0.0 211:19 0 portsentry
13237 apache 16 0 0 0 0 Z 0.7 0.0 0:00 0 httpd <defunc
2 root 15 0 0 0 0 SW 0.1 0.0 0:02 0 keventd
14103 apache 21 0 444 444 384 S 0.1 0.0 0:00 0 m
1 root 15 0 108 80 56 S 0.0 0.0 0:08 0 init
3 root 34 19 0 0 0 SWN 0.0 0.0 0:13 0 ksoftirqd_CPU
8 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 bdflush
4 root 15 0 0 0 0 RW 0.0 0.0 3:42 0 kswapd
5 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kscand/DMA
6 root 16 0 0 0 0 SW 0.0 0.0 63:06 0 kscand/Normal
7 root 16 0 0 0 0 SW 0.0 0.0 345:58 0 kscand/HighMe
9 root 15 0 0 0 0 SW 0.0 0.0 0:03 0 kupdated
10 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd

Let's see if I can add something to this thread :)

In my opinion, once he posted the memory usage of the apache processes, the root of the problem was likely found. I had a look at my apache processes, and I'm using less than 4mb memory on each process. In his paste, he's using between 6 and 9mb for each process.. which is insane. The problem is the disk i/o from running out of memory, and I doubt there is much (if any) problem with the cpu being able to keep up.

I'm guessing, like someone else had mentioned, that you're loading a ton more modules than you're actually using. That's where I'd start looking. Also, I'd venture to guess that apache 2.x uses more memory than the 1.3 tree, which is what I'm using. Something else I'd try, if you have the means, is to download, compile, and install the newest apache 1.3.x to see if you get better performance. The configs are essentially the same, so there'd probably be only minor tweaking to get the test running. (I don't know exactly what you're doing, so I can't say for certain what would or would not need changed :))

Even further, I saw where you had the max clients set to "300" which is too high for how much ram you're using on each process. If you're using between 6 and 9mb memory, let's assume 8 for this little bit of math. You're supposed to divide your total ram (2048mb) minus system overhead, which I'll just be simple and assume 32mb, by the amount of memory each process takes up (8mb). That leaves us with this

(2048 - 32) / 8 = 252

To be even a little more cautious, I'd set max clients to 250. This will avoid having apache to swap out, because the maximum amount of apache processes won't exceed your available ram.

Alright, so that's a little summary of what has already been said, with a little more of my own experience. If you'd like, I can take a look at your machine free of charge. It's 5am, and I'm bored :)

Paul,

I know quite a few people who have been burned by computer consultants, there are numerous charlatans amongst the computer trade. trying to make a fast buck.

You might want to approach your future services purchases differently:
contingency, paid only if an objective is achieved.

progress payments, if x is achieved I pay 10% of the fee, once Z is done I pay another 10% , likewise this ensures you only pay if they make progress.

Outline your maximum fee, and state the amount of hours you are paying for, I just heard from a person we did business with, a company came out to see why the backup tape only went to 75% before completion, two consultants stayed on site, watching the tape do the backup for five hours, billed for 10 man hours to "watch" the backup? personally I wouldn't pay that bill.
If you outline your maximum fee/hours in advance there is no "pumping" the bill with more hours/excessive hours.

and finally all of the above, in writing, and agreed upon by both parties.

good luck!

btw. Jason the routing admin @ He.net freemont location is quite knowledgeable you might ask him about helping you streamline your apache configs etc.

Tough to say, could be too many run away processes, harvesting your content, poor content delivery system or bad nic. Good luck bro