SplitInfinity

I am posting this because there are alot of people suffering from this
hard to identify problem. So I researched it and came up with the solution.
My gift to the universe.

Is your server eating up memory for apparantly no reason?
Does your memory usage continually grow in 4096 byte increments (approx.)
and fill up all available RAM then being using swap space until it's full
as well? Is your servers load average like 900 or some crazy high number?

The culprit of your problem may be as simple as the kernel version you
are using.

Everyone who had this problem that I helped was using kernel version:
2.6.14, however it might affect other releases as well.

In my research, I have found that this kernel release had a big bad memory
leak related to buggy ext3 filesystem code.

Your server will appear ok until you start up apache or mysql or anything that uses the hard drive. The problem appears to manifest itself as a
side afect of apache (because it shows up typically only when apache is running, which is misleading). Techs will immediatly jump to apache and try solving the problem by tuning apache's MaxClients and Keep Alives and MaxRequestsPerChild.... to no avail. This is because there are settings in
Apache to combat "memory leaks", which are actually not for linux systems
and won't help in this situation.

The buggy ext3 filesystem code in the 2.6.14 Kernel eats up memory
during disk usage. On a lightly used server, the problem may take
weeks to show up (do you have to reboot your server every week or so?).

On a heavily used server, it may take all but 30 minutes of usage. The rule of thumb is the more you use the hard drives, the quicker it eats up the memory, and the quicker your load goes out of control due to Virtual Memory Swap Thrashing.

Your load goes high because of the swap space thrashing and swapping pages continually in and out of memory. This is what causes the load, not apache or mysql.

The solution? Upgrade to kernel 2.6.16, reboot and enjoy the new relaxed attitude your server will have.

Happy Memorial Day everyone!
-SpLiTiNfInItY-