PHP, Database, Server admin experts.. Got a question - GoFuckYourself.com

lucky482 · 11-17-2007, 01:07 AM

When a tgp script crops and save thumbnail images, should it....

1. Save it all to one folder? Like this...
www.domain.com/thumbs/imageid.jpg
www.domain.com/thumbs/imageid.jpg
www.domain.com/thumbs/imageid.jpg

or

2. Save it into image id subfolders? Like this...
www.domain.com/thumbs/imageid/1.jpg
www.domain.com/thumbs/imageid/1.jpg
www.domain.com/thumbs/imageid/1.jpg

Which is better for output, PHP, database, and server performance? Or does it even make a difference?

Thanks in advance!

TidalWave · 11-17-2007, 01:19 AM

if you plan to have over 30,000 images you will want to do it option #2

moses · 11-17-2007, 01:24 AM

If you have a lot of images, option #2 for sure, ideally even splitting it up so that for image123.jpg it's images/1/2/123.jpg or something similar.

If you have too many images (say, 1000 as a nice round number) per directory, you're going to start incurring performance hits.

GrouchyAdmin · 11-17-2007, 01:27 AM

Quote:

Originally Posted by TidalWave

if you plan to have over 30,000 images you will want to do it option #2

Yep. Even directory hashing can only do so much... and god forbid doing a full lookup.

Evil E · 11-17-2007, 01:28 AM

At what point should it become a problem(in terms of 10k thumbs increments) if you have a script that does it option 1 style?

2HousePlague · 11-17-2007, 01:29 AM

It's good to see that there are still technical, webmastery type threads here on GFY. I can't tell you how I much I learned here. Truly, I was and remain the noob.

justFred · 11-17-2007, 01:33 AM

you should start with the system you intend to use all along, IE option #2. If you go with option #1 eventually you're going to have to change the system, which is more work than you originally needed to do.

lucky482 · 11-17-2007, 01:39 AM

Thanks! I guess I am going to go with option #2.

Evil E · 11-17-2007, 01:40 AM

Quote:

Originally Posted by 2HousePlague

It's good to see that there are still technical, webmastery type threads here on GFY. I can't tell you how I much I learned here. Truly, I was and remain the noob.

They are pretty rare nowadays, but I always hop in to see if there's anything interesting going on.

TidalWave · 11-17-2007, 01:44 AM

Quote:

Originally Posted by knew

At what point should it become a problem(in terms of 10k thumbs increments) if you have a script that does it option 1 style?

at 30,000 files in 1 directory the OS limit becomes the problem and you will need to start a new directory.
not to mention querying a directory with 30,000 (or 10,000 even) files would take forever.

GrouchyAdmin · 11-17-2007, 01:46 AM

Another good option is to use is date hashing, so if you want to clean things out, it's quite simple.

/YYYY/MM/DD/thumb... which makes the linking a little uglier, but it's a much better solution than
/firstletterornumber/second/thumb - from the standpoint of if you want to clean things up and eventually remove them.

Evil E · 11-17-2007, 01:53 AM

Quote:

Originally Posted by TidalWave

at 30,000 files in 1 directory the OS limit becomes the problem and you will need to start a new directory.
not to mention querying a directory with 30,000 (or 10,000 even) files would take forever.

I will look into this as this one script I use seems to keep all it's crap into the same dir... I will check to see if there is any option to change that.

thumbs]$ ls -al|wc -l
6890

lucky482 · 11-17-2007, 02:44 AM

Quote:

Originally Posted by knew

I will look into this as this one script I use seems to keep all it's crap into the same dir... I will check to see if there is any option to change that.

thumbs]$ ls -al|wc -l
6890

Yeah that was one of the main reasons why I asked because a couple of scripts used option #1 but usually the better/popular scripts use option #2

lucky482 · 11-17-2007, 02:47 AM

Quote:

Originally Posted by GrouchyAdmin

Another good option is to use is date hashing, so if you want to clean things out, it's quite simple.

/YYYY/MM/DD/thumb... which makes the linking a little uglier, but it's a much better solution than
/firstletterornumber/second/thumb - from the standpoint of if you want to clean things up and eventually remove them.

That's an idea, but I don't want to associate the date with the location folders just in case I rotate gallery display dates, I might get confused

lucky482 · 11-17-2007, 02:49 AM

Quote:

Originally Posted by 2HousePlague

It's good to see that there are still technical, webmastery type threads here on GFY. I can't tell you how I much I learned here. Truly, I was and remain the noob.

Usually those type of threads are made by guys with old registration dates but low post count.

lucky482 · 11-17-2007, 05:11 AM

Quote:

Originally Posted by pornopete

The first question is what OS will you be working on? Windows, Linux or BSD.

Even better do you know the file system you are using? NTFS, EXT3, HFS+, UFS, FAT32

You shouldn't have a problem loading them all in one folder unless you are using an older file system like FAT32 which will only let you store 65,534 files or folders per folder. In which case neither one of you proposed solutions would work.

As for which is faster for output it really depends on what are outputting.

Well the server is FreeBSD. As for the file system, I'm not sure but I know it probably not an older system like FAT32. As for the output, it will be categorized thumb preview gallery pages.

lucky482 · 11-17-2007, 05:16 AM

Quote:

Originally Posted by TidalWave

at 30,000 files in 1 directory the OS limit becomes the problem and you will need to start a new directory.
not to mention querying a directory with 30,000 (or 10,000 even) files would take forever.

So is it ok to have over 30,000 folders in 1 directory or do folders count has files too?

TidalWave · 11-17-2007, 05:24 AM

folders are fine

Zorgman · 11-17-2007, 05:59 AM

Good thread, helped me with a recent problem on my tube script. Thanks guys.

lucky482 · 11-17-2007, 07:01 AM

Quote:

Originally Posted by TidalWave

folders are fine

OK thanks for the information. So it's ok then to have lets say 2,000,000 subfolders in one directory as long as not too many files in each subfolders.

Quote:

Originally Posted by pornopete

You are probably using UFS2 in which case you should be fine using 1 folder. I believe the only limitation is that the folder cannot exceed 1 YiB in size but you might want to double check that.

I read somewhere the limit is 1,000,000 files per folder but I think the guy who posted that is wrong.

Probably go with option 2 then because can't take the risk of only 1,000,000 file limit. With about a million fhg out there and cropping thumbnails into different sizes and stuff, image count would be way over a million.

Thanks again guys, gotta let my programmer know.

Evil E · 11-17-2007, 09:50 AM

The official answer from the staff at JMB, what do you think?

Quote:

Originally Posted by JMB-staff

At most this should only cause a very minimal affect on your server. I have seen AutoGallery SQL and TGPX installations that have over 100,000 thumbnails and are running without any problems.

11-17-2007, 01:28 AM	#5
Evil E Confirmed User Join Date: Apr 2005 Location: Lazyness is a lifestyle Posts: 3,201	At what point should it become a problem(in terms of 10k thumbs increments) if you have a script that does it option 1 style? __________________ A girl once told me "Give me 8 inches and make it HURT". So, I fucked her twice and hit her with a brick.

11-17-2007, 01:29 AM	#6
2HousePlague CURATOR Join Date: Jul 2004 Location: the attic Posts: 14,572	It's good to see that there are still technical, webmastery type threads here on GFY. I can't tell you how I much I learned here. Truly, I was and remain the noob. __________________ tada!

11-17-2007, 01:33 AM	#7
justFred Confirmed User Join Date: Mar 2007 Posts: 922	you should start with the system you intend to use all along, IE option #2. If you go with option #1 eventually you're going to have to change the system, which is more work than you originally needed to do. __________________ Vote Bill Cosby 2012

11-17-2007, 01:46 AM	#11
GrouchyAdmin Now choke yourself! Industry Role: Join Date: Apr 2006 Posts: 12,085	Another good option is to use is date hashing, so if you want to clean things out, it's quite simple. /YYYY/MM/DD/thumb... which makes the linking a little uglier, but it's a much better solution than /firstletterornumber/second/thumb - from the standpoint of if you want to clean things up and eventually remove them. __________________ AIM: GrouchyGfy

11-17-2007, 05:24 AM	#18
TidalWave Confirmed User Industry Role: Join Date: Sep 2007 Location: Los Angeles Posts: 2,706	folders are fine __________________ www.SwiftNode.com

11-17-2007, 01:07 AM	#1
lucky482 Confirmed User Join Date: Mar 2002 Location: Los Angeles Posts: 252	PHP, Database, Server admin experts.. Got a question When a tgp script crops and save thumbnail images, should it.... 1. Save it all to one folder? Like this... www.domain.com/thumbs/imageid.jpg www.domain.com/thumbs/imageid.jpg www.domain.com/thumbs/imageid.jpg or 2. Save it into image id subfolders? Like this... www.domain.com/thumbs/imageid/1.jpg www.domain.com/thumbs/imageid/1.jpg www.domain.com/thumbs/imageid/1.jpg Which is better for output, PHP, database, and server performance? Or does it even make a difference? Thanks in advance!

11-17-2007, 01:19 AM	#2
TidalWave Confirmed User Industry Role: Join Date: Sep 2007 Location: Los Angeles Posts: 2,706	if you plan to have over 30,000 images you will want to do it option #2 __________________ www.SwiftNode.com

11-17-2007, 01:24 AM	#3
moses Confirmed User Join Date: Aug 2002 Posts: 56	If you have a lot of images, option #2 for sure, ideally even splitting it up so that for image123.jpg it's images/1/2/123.jpg or something similar. If you have too many images (say, 1000 as a nice round number) per directory, you're going to start incurring performance hits.

11-17-2007, 01:39 AM	#8
lucky482 Confirmed User Join Date: Mar 2002 Location: Los Angeles Posts: 252	Thanks! I guess I am going to go with option #2.

11-17-2007, 05:59 AM	#19
Zorgman Confirmed User Join Date: Aug 2002 Location: Sydney, Australia Posts: 6,103	Good thread, helped me with a recent problem on my tube script. Thanks guys. __________________ ---