PHP, Database, Server admin experts.. Got a question

[lucky482]

When a tgp script crops and save thumbnail images, should it....

1. Save it all to one folder? Like this...
www.domain.com/thumbs/imageid.jpg
www.domain.com/thumbs/imageid.jpg
www.domain.com/thumbs/imageid.jpg

or

2. Save it into image id subfolders? Like this...
www.domain.com/thumbs/imageid/1.jpg
www.domain.com/thumbs/imageid/1.jpg
www.domain.com/thumbs/imageid/1.jpg

Which is better for output, PHP, database, and server performance? Or does it even make a difference?

Thanks in advance!

[TidalWave]

if you plan to have over 30,000 images you will want to do it option #2

[moses]

If you have a lot of images, option #2 for sure, ideally even splitting it up so that for image123.jpg it's images/1/2/123.jpg or something similar.

If you have too many images (say, 1000 as a nice round number) per directory, you're going to start incurring performance hits.

[GrouchyAdmin]

if you plan to have over 30,000 images you will want to do it option #2

Yep. Even directory hashing can only do so much... and god forbid doing a full lookup.

[Evil E]

At what point should it become a problem(in terms of 10k thumbs increments) if you have a script that does it option 1 style?

[2HousePlague]

It's good to see that there are still technical, webmastery type threads here on GFY. I can't tell you how I much I learned here. Truly, I was and remain the noob.

[justFred]

you should start with the system you intend to use all along, IE option #2. If you go with option #1 eventually you're going to have to change the system, which is more work than you originally needed to do.

[lucky482]

Thanks! I guess I am going to go with option #2.

[Evil E]

It's good to see that there are still technical, webmastery type threads here on GFY. I can't tell you how I much I learned here. Truly, I was and remain the noob.

They are pretty rare nowadays, but I always hop in to see if there's anything interesting going on.

[TidalWave]

At what point should it become a problem(in terms of 10k thumbs increments) if you have a script that does it option 1 style?

at 30,000 files in 1 directory the OS limit becomes the problem and you will need to start a new directory.
not to mention querying a directory with 30,000 (or 10,000 even) files would take forever.

[GrouchyAdmin]

Another good option is to use is date hashing, so if you want to clean things out, it's quite simple.

/YYYY/MM/DD/thumb... which makes the linking a little uglier, but it's a much better solution than
/firstletterornumber/second/thumb - from the standpoint of if you want to clean things up and eventually remove them.

[Evil E]

at 30,000 files in 1 directory the OS limit becomes the problem and you will need to start a new directory.
not to mention querying a directory with 30,000 (or 10,000 even) files would take forever.

I will look into this as this one script I use seems to keep all it's crap into the same dir... I will check to see if there is any option to change that.

thumbs]$ ls -al|wc -l
6890

[lucky482]

I will look into this as this one script I use seems to keep all it's crap into the same dir... I will check to see if there is any option to change that.

thumbs]$ ls -al|wc -l
6890

Yeah that was one of the main reasons why I asked because a couple of scripts used option #1 but usually the better/popular scripts use option #2

[lucky482]

Another good option is to use is date hashing, so if you want to clean things out, it's quite simple.

/YYYY/MM/DD/thumb... which makes the linking a little uglier, but it's a much better solution than
/firstletterornumber/second/thumb - from the standpoint of if you want to clean things up and eventually remove them.

That's an idea, but I don't want to associate the date with the location folders just in case I rotate gallery display dates, I might get confused

[lucky482]

It's good to see that there are still technical, webmastery type threads here on GFY. I can't tell you how I much I learned here. Truly, I was and remain the noob.

Usually those type of threads are made by guys with old registration dates but low post count.

[lucky482]

The first question is what OS will you be working on? Windows, Linux or BSD.

Even better do you know the file system you are using? NTFS, EXT3, HFS+, UFS, FAT32

You shouldn't have a problem loading them all in one folder unless you are using an older file system like FAT32 which will only let you store 65,534 files or folders per folder. In which case neither one of you proposed solutions would work.

As for which is faster for output it really depends on what are outputting.

Well the server is FreeBSD. As for the file system, I'm not sure but I know it probably not an older system like FAT32. As for the output, it will be categorized thumb preview gallery pages.

[lucky482]

at 30,000 files in 1 directory the OS limit becomes the problem and you will need to start a new directory.
not to mention querying a directory with 30,000 (or 10,000 even) files would take forever.

So is it ok to have over 30,000 folders in 1 directory or do folders count has files too?

[TidalWave]

folders are fine

[Zorgman]

Good thread, helped me with a recent problem on my tube script. Thanks guys.

[lucky482]

folders are fine

OK thanks for the information. So it's ok then to have lets say 2,000,000 subfolders in one directory as long as not too many files in each subfolders.

You are probably using UFS2 in which case you should be fine using 1 folder. I believe the only limitation is that the folder cannot exceed 1 YiB in size but you might want to double check that.

I read somewhere the limit is 1,000,000 files per folder but I think the guy who posted that is wrong.

Probably go with option 2 then because can't take the risk of only 1,000,000 file limit. With about a million fhg out there and cropping thumbnails into different sizes and stuff, image count would be way over a million.

Thanks again guys, gotta let my programmer know.

[Evil E]

The official answer from the staff at JMB, what do you think?

At most this should only cause a very minimal affect on your server. I have seen AutoGallery SQL and TGPX installations that have over 100,000 thumbnails and are running without any problems.