Quick Question: How many files can you have in a Linux Server folder? - GoFuckYourself.com

$5 submissions · 03-21-2005, 04:11 PM

I'm building thousands of pages but the way the script is designed, all the html files must be in one directory. Anyone know if there's a limit in Linux re how many files can be in a directory?

Jace · 03-21-2005, 04:25 PM

i have had upwards of 15,000 before...never had a problem

Nembrionic · 03-21-2005, 04:33 PM

http://www.google.com/search?hl=en&q...irectory+linux

will give this as one of the pages:
http://www.uwsg.iu.edu/hypermail/lin...05.0/0164.html

now that was hard! :D

Nembrionic · 03-21-2005, 04:34 PM

Ofcourse it depends on your FS and maybe even your linux flavor

$5 submissions · 03-21-2005, 04:50 PM

Quote:

Originally Posted by JaceXXX

i have had upwards of 15,000 before...never had a problem

Thanks, Jace, much appreciated.

mortenb · 03-21-2005, 05:10 PM

I recently saw a system getting unstable at around 30k pics in a folder

toddler · 03-21-2005, 05:16 PM

it depends more on how your server parses the files vs just the raw counts. For longterm managability, use sub dirs. scripts parsing many thousands of files in a single directory could very well have problems with the size of the arrays. Use sub dirs...

azguy · 03-21-2005, 05:17 PM

Depends on the Linux distro and kernel version, but I don't think you'll have a problem any time soon

fuzebox · 03-21-2005, 05:20 PM

The limit under ext3 is 32768 files/subdirectorys in one directory

zagi · 03-21-2005, 05:30 PM

The limits change from one OS to another, but one thing is true, Unix + Linux is notoriously bad with a large number of files in one directory.

I would have to estimate the upper limit between 20,000 - 30,000 after that performance suffers heavily. Yes the script may not use sub directories but it can always be modified. Usuaully you wont notice until its too late and then rewriting + sorting archived files is a big pain.

Take this into account before going production with this system would be my advice.

Under FreeBSD you can tune the following sysctl variables to increase performance for large # of files in a directory:

vfs.ufs.dirhash_minsize: 2560
vfs.ufs.dirhash_maxmem: 2097152

swedguy · 03-21-2005, 05:41 PM

32,768 for both ext2 and ext3. I found that out the hard way with an old redirecting service

goBigtime · 03-21-2005, 05:48 PM

If you have to ask that question, then you should probably rework your application to use subdirs. Dealing with directories with that many files can waste a lot of resources.

The easiest way might be to create folders named 0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z and rewrite your script to seek/store the files in that manner.

But it really depends on your application and file naming scheme.

esteve · 03-21-2005, 05:50 PM

I have about 150,000 files under a folder in windows, No they're not all in one huge line but /asian/10/ /anal/01/
dunno if that helps

03-21-2005, 04:11 PM	#1
$5 submissions I help you SUCCEED Industry Role: Join Date: Nov 2003 Location: The Pearl of the Orient Seas Posts: 32,195	Quick Question: How many files can you have in a Linux Server folder? I'm building thousands of pages but the way the script is designed, all the html files must be in one directory. Anyone know if there's a limit in Linux re how many files can be in a directory? __________________ Need a NON-ADULT income stream? We build your YOUTUBE CHANNEL for only $20 per video! For Hire: $3/hour or $400 per month Marketing Specialist Virtual Assistants

03-21-2005, 05:16 PM	#7
toddler Confirmed User Join Date: Jun 2002 Location: austin, tx Posts: 1,911	it depends more on how your server parses the files vs just the raw counts. For longterm managability, use sub dirs. scripts parsing many thousands of files in a single directory could very well have problems with the size of the arrays. Use sub dirs... __________________ http://www.flickr.com/photos/zoddler/

03-21-2005, 05:30 PM	#10
zagi Confirmed User Join Date: Jan 2004 Posts: 1,238	The limits change from one OS to another, but one thing is true, Unix + Linux is notoriously bad with a large number of files in one directory. I would have to estimate the upper limit between 20,000 - 30,000 after that performance suffers heavily. Yes the script may not use sub directories but it can always be modified. Usuaully you wont notice until its too late and then rewriting + sorting archived files is a big pain. Take this into account before going production with this system would be my advice. Under FreeBSD you can tune the following sysctl variables to increase performance for large # of files in a directory: vfs.ufs.dirhash_minsize: 2560 vfs.ufs.dirhash_maxmem: 2097152 __________________ Managed US/NL Hosting [ [Reality Check Network ] Dell XEON Servers + 1/2/3 TB Packages ICQ: 4-930-562

03-21-2005, 05:50 PM	#13
esteve Confirmed User Join Date: May 2004 Location: Whistler, B.C. Canada Posts: 233	I have about 150,000 files under a folder in windows, No they're not all in one huge line but /asian/10/ /anal/01/ dunno if that helps __________________ * JayManCash * Raven Riley * Liz Vicious Toni KatVixen Bangin Becky * * Duke Dollars * Facial Abuse * Ghetto Gaggers *

03-21-2005, 04:25 PM	#2
Jace FBOP Class Of 2013 Industry Role: Join Date: Jan 2004 Location: bumfuck, ky Posts: 35,562	i have had upwards of 15,000 before...never had a problem

03-21-2005, 04:33 PM	#3
Nembrionic Confirmed User Join Date: Feb 2003 Posts: 2,424	http://www.google.com/search?hl=en&q...irectory+linux will give this as one of the pages: http://www.uwsg.iu.edu/hypermail/lin...05.0/0164.html now that was hard! :D

03-21-2005, 04:34 PM	#4
Nembrionic Confirmed User Join Date: Feb 2003 Posts: 2,424	Ofcourse it depends on your FS and maybe even your linux flavor

03-21-2005, 05:10 PM	#6
mortenb Confirmed User Join Date: Jul 2004 Location: Denmark ICQ: 7880009 Posts: 2,203	I recently saw a system getting unstable at around 30k pics in a folder

03-21-2005, 05:17 PM	#8
azguy Confirmed User Join Date: Nov 2004 Location: Scottsdale, AZ Posts: 5,167	Depends on the Linux distro and kernel version, but I don't think you'll have a problem any time soon

03-21-2005, 05:20 PM	#9
fuzebox making it rain Industry Role: Join Date: Oct 2003 Location: seattle Posts: 22,119	The limit under ext3 is 32768 files/subdirectorys in one directory

03-21-2005, 05:41 PM	#11
swedguy Confirmed User Industry Role: Join Date: Jan 2002 Posts: 7,981	32,768 for both ext2 and ext3. I found that out the hard way with an old redirecting service

03-21-2005, 05:48 PM	#12
goBigtime Confirmed User Join Date: Nov 2002 Posts: 7,761	If you have to ask that question, then you should probably rework your application to use subdirs. Dealing with directories with that many files can waste a lot of resources. The easiest way might be to create folders named 0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z and rewrite your script to seek/store the files in that manner. But it really depends on your application and file naming scheme.