Copying 10 Million Thumbs, Fuck!

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • PowerCum
    CjOverkill
    • Apr 2003
    • 1328

    #16
    You have several options to speed this by an order of magnitude:

    Option 1:
    If both HDs are the same size and from the same company and are the same model... I mean, both HDs are the same, then you have another option that will be considerably faster.

    We use dd here instead of copying.

    dd bs=10240k if=/dev/original of=/dev/copy


    Option 2:
    If on Linux, another option is to remount the original and the target partition with noatime option and then use the usual copy or rsync as normal. This will speed things alot when copying lots of small files. In this case rsync is better, even that it will be a bit slower than cp. The advantage of rsync when copying that much info is visible if the target HD has a sector error and copy process does not work properly. Rsync will warn you while cp will not.


    Option 3:
    Setup a RAID1 software raid between both HDs and it will automatically mirror the information. It will do practically the same as in option 1, but will do it slower and you will still be able to use that HD to serve thumbs or do whatever you want with them.
    CjOverkill Traffic Trading Script
    Free, secure and fast traffic trading script. Get your copy now

    Comment

    • fris
      Too lazy to set a custom title
      • Aug 2002
      • 55679

      #17
      DD would be the quickest
      Since 1999: 69 Adult Industry awards for Best Hosting Company and professional excellence.

      Comment

      • DarkJedi
        No Refunds Issued.
        • Feb 2001
        • 28301

        #18
        That's a big tube

        Comment

        • seeandsee
          Check SIG!
          • Mar 2006
          • 50945

          #19
          Must be some solution for such kind of problem, something to finish it in 1 day
          BUY MY SIG - 50$/Year

          Contact here

          Comment

          • Hermes
            Confirmed User
            • Oct 2010
            • 264

            #20
            It's a flaw in design of file system or copy function of OS. Maybe some program can speed it up that could have raw disk write access and cache the data properly.
            The Most extensive list of Paxum affiliate programs -> Paxum Sponsors .net

            Comment

            Working...