Here's an unexpected mind fuck... 10 million thumbs, 500 gigs, from one drive to another on the same server... estimated time, 4 fucking days!
-
-
would have been much quicker to zip them then transfer the zip.
For coding work - hit me up on andy // borkedcoder // com
(consider figuring out the email as test #1)
All models are wrong, but some are useful. George E.P. Box. p202 -
i tried that first, copied all in just a few hours, but then needed 9 days to extract.. so i deleted the gzip and started over with copyComment
-
my second attempt has been running 13 hours now.. no compression sounds interesting, but not sure if the performance gain will be worth starting over now.. with compression was 9 days to extract.. without, maybe half? which would be the same as copy then...Comment
-
WTF, one of your thumbs is as big as 50MB? Normally a thumb is below 10kB.
Comment
-
160px × 120px, average 15 KB each.. in this case it's not the size, it's the number.Comment
-
-
Comment
-
Comment
-
i'm guesstimating... roughly 500k sets of 20 thumbs each, filled a 500gb drive 92%.. some thumbs are 10kb, some are 35kb.. and roughly 10-million of them..
edit.. was off by 172,000 set.. total is 672k sets for 20Last edited by rickdu; 02-26-2012, 12:37 AM.Comment
-
windows or unix?Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence.
Comment
-
unix.. freebsd 8Comment
-
Use coupon 'pauljohn' for a $1 discount at already super cheap NameSilo!
Anal Webcams | Kinky Trans Cams Live | Hotwife XXX Tube | Get your Proxies hereComment
-
You have several options to speed this by an order of magnitude:
Option 1:
If both HDs are the same size and from the same company and are the same model... I mean, both HDs are the same, then you have another option that will be considerably faster.
We use dd here instead of copying.
dd bs=10240k if=/dev/original of=/dev/copy
Option 2:
If on Linux, another option is to remount the original and the target partition with noatime option and then use the usual copy or rsync as normal. This will speed things alot when copying lots of small files. In this case rsync is better, even that it will be a bit slower than cp. The advantage of rsync when copying that much info is visible if the target HD has a sector error and copy process does not work properly. Rsync will warn you while cp will not.
Option 3:
Setup a RAID1 software raid between both HDs and it will automatically mirror the information. It will do practically the same as in option 1, but will do it slower and you will still be able to use that HD to serve thumbs or do whatever you want with them.Comment
-
DD would be the quickestSince 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence.
Comment
-
That's a big tube
Comment
-
Must be some solution for such kind of problem, something to finish it in 1 dayComment
-
It's a flaw in design of file system or copy function of OS. Maybe some program can speed it up that could have raw disk write access and cache the data properly.Comment
Comment