Backup solutions for big sites - GoFuckYourself.com - Adult Webmaster Forum

X

strobi

Confirmed User

Join Date: Nov 2002

Posts: 7383
Share

Post
#1

Backup solutions for big sites

04-20-2010, 10:34 AM

With more and more tubes popping up with terrabytes of data. How do these guys manage the backups?

RAID isn't an option, since drive failures/ drive copies always can go wrong.

Offsite backups are pain in the ass to move 10+TB every day at like 10 mbits...

Incremental backups could help, but copying the whole system from time to time is needed and runs in the two digits or more TBs.

How do the big hosting firms do those needed backups of clients with LOADS of needed disc space?

Anyway, discuss...
Tags: None
Barefootsies

Choice is an Illusion

Join Date: Feb 2005

Posts: 42635
Share

Post
#2

04-20-2010, 10:38 AM

Actually that's why I do RAID on all my boxes PLUS secondary back ups.

RAID should save you if your main hard drive fails. So your site is not down when you are doing a hardware swap out. Then you have a secondary system in place for your daily/weekly/monthly.

As for myself personally, I also have back ups in my office of websites and their content. So should all else fail, I could FedEx to host/datcenter/office and have them put it on their server at the office.

Last edited by Barefootsies; 04-20-2010, 10:42 AM.

Should You Email Your Members?

Link1 | Link2 | Link3

Enough Said.

"Would you rather live like a king for a year or like a prince forever?"
Comment
DateDoc

Outside looking in.

Join Date: Feb 2005

Posts: 14243
Share

Post
#3

04-20-2010, 10:39 AM

I'd think most of the bigger sites use a CDN which has all their content on multiple servers worldwide. For all those to fail seems very unlikely. That leaves a much smaller backup - db, etc.
Comment
strobi

Confirmed User

Join Date: Nov 2002

Posts: 7383
Share

Post
#4

04-20-2010, 10:45 AM

Originally posted by DateDoc

I'd think most of the bigger sites use a CDN which has all their content on multiple servers worldwide. For all those to fail seems very unlikely. That leaves a much smaller backup - db, etc.

I guess this is one of the only options. What puzzles me how you can make a backup of a 20 TB site for example?

Just copying the files from A to B to do the backup will take a longgggggggggg time.

Cloud hosting would be a solotion, if the cloud is really robust.

Small sites you can always just make a tar/zip file and send it to another hoster/server. But doing this with 10TB+, madness!
Comment
strobi

Confirmed User

Join Date: Nov 2002

Posts: 7383
Share

Post
#5

04-20-2010, 10:48 AM

Originally posted by Barefootsies

Actually that's why I do RAID on all my boxes PLUS secondary back ups.

RAID should save you if your main hard drive fails. So your site is not down when you are doing a hardware swap out. Then you have a secondary system in place for your daily/weekly/monthly.

As for myself personally, I also have back ups in my office of websites and their content. So should all else fail, I could FedEx to host/datcenter/office and have them put it on their server at the office.

Great backup plan you got there!

But doing RAID and daily/weekly, if you are sporting HUGE amounts of data?

How does that work?

Leaves me puzzled...

RAID with 20 drives would be doable, but backing that up :-(
Comment
DateDoc

Outside looking in.

Join Date: Feb 2005

Posts: 14243
Share

Post
#6

04-20-2010, 10:51 AM

Originally posted by strobi

I guess this is one of the only options. What puzzles me how you can make a backup of a 20 TB site for example?

Just copying the files from A to B to do the backup will take a longgggggggggg time.

Cloud hosting would be a solotion, if the cloud is really robust.

Small sites you can always just make a tar/zip file and send it to another hoster/server. But doing this with 10TB+, madness!

Imagine how long it takes to upload 20TB and populate the CDN
Comment
Barefootsies

Choice is an Illusion

Join Date: Feb 2005

Posts: 42635
Share

Post
#7

04-20-2010, 10:54 AM

Originally posted by strobi

Great backup plan you got there!

But doing RAID and daily/weekly, if you are sporting HUGE amounts of data?

How does that work?

Leaves me puzzled...

RAID with 20 drives would be doable, but backing that up :-(

Well, it sounds like you have more data then I do in that regard. But I do video heavy sites myself personally. So I feel your pain. Not to the 20TB degree, but I can relate.

Thanks on the kudos. Yeah, I try and make sure myself, and clients, are never going to be down. It costs more to do things this way with the RAID ARRAY. But it does make sure that if a hard drive fails, I do not need to worry about ordering a HD. I can do a simple swap, and tada. Client remains online while getting failed drive replaced.

Should You Email Your Members?

Link1 | Link2 | Link3

Enough Said.

"Would you rather live like a king for a year or like a prince forever?"
Comment
KingRedOnion

Confirmed User

Join Date: Feb 2007

Posts: 189
Share

Post
#8

04-20-2010, 10:57 AM

Mirrored SAN with hourly, daily and monthly snapshots.

Trade Adult - 250+ TGPs looking for trades
Comment
mikke

Confirmed User

Join Date: Jan 2010

Posts: 1327
Share

Post
#9

04-20-2010, 11:00 AM

rsync + storage box in ovh.com, thats all

icq: 395 294 346
http://www.adultsubmitter.eu - submit any adult site to 20 directories from 1 form!
now 20 domains!
http://www.porndeals.eu http://www.ebonybangbros.com
Comment
chupachups

Confirmed User

Join Date: Dec 2002

Posts: 6576
Share

Post
#10

04-20-2010, 11:04 AM

2 online boxes with 2x750gb
50gb at one host
50gb at another host
1 online backup service on one of those boxes
1 backup server with backup service at home (keepit)
sending a usb memory to a friend every now and then
gmail
dvd and hard drive in safe
hidden 2.5 wireless NAS

Damn
Comment
strobi

Confirmed User

Join Date: Nov 2002

Posts: 7383
Share

Post
#11

04-20-2010, 11:08 AM

Originally posted by mikke

rsync + storage box in ovh.com, thats all

Was looking into that too, they have super cheap 24 drive setups, but filling that thing over even a 1000 mbit line would take some time :-)
Comment
fpforum

Confirmed User

Join Date: Dec 2006

Posts: 1045
Share

Post
#12

04-20-2010, 11:13 AM

RAID is great for one..Also, remote backups are highly recommended. Make sure you are keeping your backups not only on your main server, but on a remote server as well!

Email: strikedata |@| gmail.com Skype: mistercashman
Comment
strobi

Confirmed User

Join Date: Nov 2002

Posts: 7383
Share

Post
#13

04-20-2010, 11:17 AM

OVH is insane! 36x 2000GB SAS array, 10 gigabit uncapped unmetered incomming, for only 1K$/month. How do they make money on that? Lol!
Comment
ScriptWorkz

Confirmed User

Join Date: Jul 2007

Posts: 274
Share

Post
#14

04-20-2010, 11:21 AM

With something like rsync you would only be moving new data or data that's changed, not everything all the time. So there ya go.

Also i do offsite backups for some of my clients, so if anyone in here need a secure / specialized offsite backup solution at a reasonable price, hit me up and i'll see how i can help. sales [at] scriptworkz.com

Last edited by ScriptWorkz; 04-20-2010, 11:23 AM.
Comment

Previous template Next

Working...