Linux Servers - Important Kernel Update - GoFuckYourself.com

SplitInfinity · 05-29-2006, 11:37 AM

I am posting this because there are alot of people suffering from this
hard to identify problem. So I researched it and came up with the solution.
My gift to the universe.

Is your server eating up memory for apparantly no reason?
Does your memory usage continually grow in 4096 byte increments (approx.)
and fill up all available RAM then being using swap space until it's full
as well? Is your servers load average like 900 or some crazy high number?

The culprit of your problem may be as simple as the kernel version you
are using.

Everyone who had this problem that I helped was using kernel version:
2.6.14, however it might affect other releases as well.

In my research, I have found that this kernel release had a big bad memory
leak related to buggy ext3 filesystem code.

Your server will appear ok until you start up apache or mysql or anything that uses the hard drive. The problem appears to manifest itself as a
side afect of apache (because it shows up typically only when apache is running, which is misleading). Techs will immediatly jump to apache and try solving the problem by tuning apache's MaxClients and Keep Alives and MaxRequestsPerChild.... to no avail. This is because there are settings in
Apache to combat "memory leaks", which are actually not for linux systems
and won't help in this situation.

The buggy ext3 filesystem code in the 2.6.14 Kernel eats up memory
during disk usage. On a lightly used server, the problem may take
weeks to show up (do you have to reboot your server every week or so?).

On a heavily used server, it may take all but 30 minutes of usage. The rule of thumb is the more you use the hard drives, the quicker it eats up the memory, and the quicker your load goes out of control due to Virtual Memory Swap Thrashing.

Your load goes high because of the swap space thrashing and swapping pages continually in and out of memory. This is what causes the load, not apache or mysql.

The solution? Upgrade to kernel 2.6.16, reboot and enjoy the new relaxed attitude your server will have.

Happy Memorial Day everyone!
-SpLiTiNfInItY-

SplitInfinity · 05-30-2006, 12:00 AM

Not much attention to this as I posted it over the holiday... bumping just for the sake of giving everyone the opprotunity to read. :-)

Cheers.

sex69 · 05-30-2006, 12:09 AM

Thanks!
Bump for this

Brujah · 05-30-2006, 12:11 AM

Thanks.. using 2.6.14 on my servers it looks like.

Jay_StandAhead · 05-30-2006, 12:23 AM

I've had a memory leak problem with Red Hat for ages... kjournald seems to be taking up all the memory

Theo The Theologian · 05-30-2006, 12:27 AM

You're such a nerd.

Great info.

babsy · 05-30-2006, 01:20 AM

2.6.8 for me. This kernel is nearly 2 years old! Fuck upgrading when it works just fine. Too many times I've had machines not come back up properly when trying to upgrade the kernel remotely.

PussyTeenies · 05-30-2006, 01:21 AM

thnx
allready recognized the problem

leg4 · 05-30-2006, 01:48 AM

Thanks very much for this post...We have had some weird server problems and this may be it.

Will have the Admin look in the a.m.

Thanks for taking your time to post this info and share it with other webmasters.

Love GFY when its useful, Hate GFY when its "would u hit it" posts...

THANKS!

SplitInfinity · 05-30-2006, 07:16 PM

You are welcome.

Anyone got some other odd problems I can help solve?
:-)

Note: I can't help with the wart on your penis. :-) LOL

toddler · 05-30-2006, 08:33 PM

Quote:

Originally Posted by SplitInfinity

You are welcome.

Anyone got some other odd problems I can help solve?
:-)

Note: I can't help with the wart on your penis. :-) LOL

can you tell me why the linux autofs code sucks as bad as it does?

SplitInfinity · 05-31-2006, 01:17 PM

"can you tell me why the linux autofs code sucks as bad as it does?"

The autofs code poorly spent its $440.00 on this school of trade, and
is therefore why it sucks bad.

http://www.themoscowtimes.com/storie...06/11/013.html

SplitInfinity · 05-31-2006, 01:20 PM

autofs (the userland part) is a little dodgy. It can get upset pretty easily with NFS mounts as can amd (the alternative, written by CSG at Imperial) is only a little better. Both are pretty huge programs for doing a simple task. I think I'll write a replacement over the weekend.

teksonline · 05-31-2006, 01:22 PM

who the heck uses linux for a webserver?

drama · 05-31-2006, 01:22 PM

Run FreeBSD n00bs

RobbieRye · 05-31-2006, 01:24 PM

Now there is a great post by a hosting company.

SplitInfinity · 05-31-2006, 02:16 PM

Thanks RobbieRye.

ne0 · 05-31-2006, 03:53 PM

Nice info Chris.

Quote:

Originally Posted by babsy

2.6.8 for me. This kernel is nearly 2 years old! Fuck upgrading when it works just fine. Too many times I've had machines not come back up properly when trying to upgrade the kernel remotely.

That's why there are so much hacked servers around.
"why update if it's working?" looks the same as "why drive a ferrari when your old car still running"

Makingcoin · 05-31-2006, 03:59 PM

Thanks for the heads up Chris.

SplitInfinity · 05-31-2006, 04:56 PM

"That's why there are so much hacked servers around.
"why update if it's working?" looks the same as "why drive a ferrari when your old car still running" "

So true. :-)

SgtStrider · 05-31-2006, 04:59 PM

thanks bro, ill keep this in mind

05-29-2006, 11:37 AM	#1
SplitInfinity Confirmed User Join Date: Dec 2002 Location: San Diego, CA Posts: 3,047	Linux Servers - Important Kernel Update I am posting this because there are alot of people suffering from this hard to identify problem. So I researched it and came up with the solution. My gift to the universe. Is your server eating up memory for apparantly no reason? Does your memory usage continually grow in 4096 byte increments (approx.) and fill up all available RAM then being using swap space until it's full as well? Is your servers load average like 900 or some crazy high number? The culprit of your problem may be as simple as the kernel version you are using. Everyone who had this problem that I helped was using kernel version: 2.6.14, however it might affect other releases as well. In my research, I have found that this kernel release had a big bad memory leak related to buggy ext3 filesystem code. Your server will appear ok until you start up apache or mysql or anything that uses the hard drive. The problem appears to manifest itself as a side afect of apache (because it shows up typically only when apache is running, which is misleading). Techs will immediatly jump to apache and try solving the problem by tuning apache's MaxClients and Keep Alives and MaxRequestsPerChild.... to no avail. This is because there are settings in Apache to combat "memory leaks", which are actually not for linux systems and won't help in this situation. The buggy ext3 filesystem code in the 2.6.14 Kernel eats up memory during disk usage. On a lightly used server, the problem may take weeks to show up (do you have to reboot your server every week or so?). On a heavily used server, it may take all but 30 minutes of usage. The rule of thumb is the more you use the hard drives, the quicker it eats up the memory, and the quicker your load goes out of control due to Virtual Memory Swap Thrashing. Your load goes high because of the swap space thrashing and swapping pages continually in and out of memory. This is what causes the load, not apache or mysql. The solution? Upgrade to kernel 2.6.16, reboot and enjoy the new relaxed attitude your server will have. Happy Memorial Day everyone! -SpLiTiNfInItY-

05-30-2006, 12:09 AM	#3
sex69 Confirmed User Join Date: Feb 2004 Posts: 671	Thanks! Bump for this __________________ HotNiches.com-Add Site Porn Sites- Submit For Adult Sites-Add URL Adult Webcams Sex Webcams Erotic Search Reality Sex Sex Videos

05-30-2006, 12:11 AM	#4
Brujah Beer Money Baron Industry Role: Join Date: Jan 2001 Location: brujah / gmail Posts: 22,157	Thanks.. using 2.6.14 on my servers it looks like. __________________ Free Premium Domain Lists and Tools at Clickmojo.com For Sale: Obscenity.com

05-30-2006, 12:23 AM	#5
Jay_StandAhead Confirmed User Industry Role: Join Date: Jul 2002 Posts: 3,103	I've had a memory leak problem with Red Hat for ages... kjournald seems to be taking up all the memory __________________ Our Programs: StandAhead \| IndieBucks \| BoyCrushCash \| Phoenixxx \| EmoProfits \| BritishBucks \| HunkMoney \| LatinoBucks Make $$$ with Gay! Lowest Minimum Payouts in the Business, Perfect Track Record, Amazing Sites

05-30-2006, 12:27 AM	#6
Theo The Theologian Confirmed User Join Date: Mar 2005 Posts: 1,099	You're such a nerd. Great info. __________________ I'm a hate-fuelled umbrella-wielding photographer with a winning smile, and a way with the ladies.

05-30-2006, 12:00 AM	#2
SplitInfinity Confirmed User Join Date: Dec 2002 Location: San Diego, CA Posts: 3,047	Not much attention to this as I posted it over the holiday... bumping just for the sake of giving everyone the opprotunity to read. :-) Cheers.

05-30-2006, 01:20 AM	#7
babsy Confirmed User Join Date: Mar 2006 Location: Cashapartment.com Posts: 282	2.6.8 for me. This kernel is nearly 2 years old! Fuck upgrading when it works just fine. Too many times I've had machines not come back up properly when trying to upgrade the kernel remotely. __________________ CASH APARTMENT V2 IS HERE! NATS POWERED, 60-70% ON ALL REBILLS, FREE CONTENT, MGPS, FHGS, RETAINING MEMBERS NETWORK! A SIMPLE NO B.S PROGRAM - ICQ 239 806 698

05-30-2006, 01:21 AM	#8
PussyTeenies Confirmed User Join Date: Feb 2005 Location: Haarlem and Amsterdam, capital of the porn world ;-) Posts: 6,496	thnx allready recognized the problem __________________ Need adult hosting? Contact us! WARM Hosting Need an IT solution? or someone to check your site and security? Nossie - IT Professional

05-30-2006, 01:48 AM	#9
leg4 Confirmed User Join Date: May 2003 Location: Texas Posts: 4,429	Thanks very much for this post...We have had some weird server problems and this may be it. Will have the Admin look in the a.m. Thanks for taking your time to post this info and share it with other webmasters. Love GFY when its useful, Hate GFY when its "would u hit it" posts... THANKS! __________________ >>> Contact me here email me here

05-30-2006, 07:16 PM	#10
SplitInfinity Confirmed User Join Date: Dec 2002 Location: San Diego, CA Posts: 3,047	You are welcome. Anyone got some other odd problems I can help solve? :-) Note: I can't help with the wart on your penis. :-) LOL

05-31-2006, 01:17 PM	#12
SplitInfinity Confirmed User Join Date: Dec 2002 Location: San Diego, CA Posts: 3,047	"can you tell me why the linux autofs code sucks as bad as it does?" The autofs code poorly spent its $440.00 on this school of trade, and is therefore why it sucks bad. http://www.themoscowtimes.com/storie...06/11/013.html

05-31-2006, 01:20 PM	#13
SplitInfinity Confirmed User Join Date: Dec 2002 Location: San Diego, CA Posts: 3,047	autofs (the userland part) is a little dodgy. It can get upset pretty easily with NFS mounts as can amd (the alternative, written by CSG at Imperial) is only a little better. Both are pretty huge programs for doing a simple task. I think I'll write a replacement over the weekend.

05-31-2006, 01:22 PM	#14
teksonline So Fucking Banned Join Date: Jan 2005 Location: At My Desk Posts: 2,904	who the heck uses linux for a webserver?

05-31-2006, 01:22 PM	#15
drama Confirmed User Join Date: Jul 2004 Posts: 847	Run FreeBSD n00bs

05-31-2006, 01:24 PM	#16
RobbieRye Registered User Join Date: Oct 2005 Location: The Middle of Nowhere Posts: 616	Now there is a great post by a hosting company.

05-31-2006, 02:16 PM	#17
SplitInfinity Confirmed User Join Date: Dec 2002 Location: San Diego, CA Posts: 3,047	Thanks RobbieRye.

05-31-2006, 03:59 PM	#19
Makingcoin Confirmed User Join Date: Aug 2002 Location: The Ditch Posts: 8,919	Thanks for the heads up Chris. __________________ www.MAKINGCOIN.com icq. 166-662-831 "Start making large coin!" Daddy I Get Paid To Be A Whore - Coming Soon

05-31-2006, 04:56 PM	#20
SplitInfinity Confirmed User Join Date: Dec 2002 Location: San Diego, CA Posts: 3,047	"That's why there are so much hacked servers around. "why update if it's working?" looks the same as "why drive a ferrari when your old car still running" " So true. :-)

05-31-2006, 04:59 PM	#21
SgtStrider Confirmed User Join Date: Oct 2005 Location: Your moms house! Posts: 1,196	thanks bro, ill keep this in mind __________________ 5% Webmaster Referrals - 45% Payouts - Exclusive Content Sgt Strider http://www.JustJassie.com [email protected] ICQ: 227-437-902