GoFuckYourself.com - Adult Webmaster Forum - View Single Post

Theo · 12-25-2001, 03:57 AM

oxcash.com
they have contact info
if it's about hosting them...kind of late

oxcash news
Posted By Josh on December 19

It's been a long, exhausting, frustrating and frighteningly expensive few weeks, but we've finally managed to overcome the load on the servers. By the looks of the board a lot of you are wondering what took so long...here's the condensed version of what's been happening..

About 4 weeks ago Bang Bus's bandwidth went from 50-60Mbps to 100Mbps. We were already adding servers so we put the new machines online and the traffic immediately jumped to 200 Mbps. At that point we started upgrading all our hardware to handle 400 Mbps..figuring that doubling what we were using would be sufficent. By the time the new servers arrived we were already doing 350Mbps. The minute the new servers went online it jumped to 630Mbps and eventually peaked at 680Mbps. In case you're wondering just how much that is...none of the hosts we've spoken to (Exodus, Level3, Global Crossing, XO and most of the other major providers) have seen anything like this. We'd been anticipating growth and were acting accordingly, but you really can't anticipate something like this.

As it stands now we have capacity for about 1Gbps and our architecture will scale up to 2Gbps by simply plugging more servers into the clusters. Static content is served by a cluster of nine machines each capable of pushing 100Mbps. The cluster should withstand failures on up to 3 machines simultaneously without noticeable slowdowns. Dynamic content is served by a 3 node cluster with each machine capable of handling the entire load if the other two were to fail for any reason. The database machine is currently the only single point of failure so it's the most powerful machine of the bunch and has built in hot swappable spare hard disks, network adapters, power supplies, and RAID controller. To try and overcome the single point of failure I've been rewriting most of the critical code to try to make everything at least continue to function during database failures. I'll also be adding a dedicated realtime backup server tomorrow that can be temporarily switched on in the event of problems. All the machines are connected to a high end load balancing switch that scales up to 2Gbps. We've got spare parts and machines on hand at Candid in case of problems.

We're working with a company that specializes in streaming video to build a scalable solution to do the Windows Media streaming...the fast forward/rewind/seek functionality. ETA is early January on that...until then the video will continue to stream from the web servers.

All planned upgrade work is now done. I'm sure there are still some oddities left to deal with, but for the most part everything should be working and pretty damn fast. Normal update schedules will be resuming Thursday.

I know the last few weeks have been trying for everyone, but you'll just have to trust me when I say it's been ten times as bad for us;-) Try staying up 48 hours to bring $50k worth of hardware online only to find out the minute you go live your bandwidth jumped another 300Mbps and your sites are still barely crawling along. We're pushing things to a whole new level and it's tough to foresee every obstacle that will arise along the way.

Now for the processors... They're just as fucked up on our end as they are for you guys. We deal with them constantly throughout the day trying to get to the bottom of things and get your problems resolved, but they hold all the cards so when we say "Jump" they say "Go fuck yourself with a rusty spatula". Not in so many words of course, but you get the idea. I'd been planning to be pretty close to completion on our own billing software, but the server nightmare has taken up all my time this month. I'll be picking up on it after Christmas and would say the tentative ETA is mid-February. I'd like to begin beta testing it by mid-January so a few of you braver souls can jump in on that if you just can't stand iBill/CCBill any longer.

That's pretty much it for the moment... I've got ~5000 emails to sort through over the next couple of days so if you've got something urgent call the office during the day or my cell (305-562-0357) after hours.

12-25-2001, 03:57 AM
Theo HAL 9000 Industry Role: Join Date: May 2001 Posts: 34,515	oxcash.com they have contact info if it's about hosting them...kind of late oxcash news Posted By Josh on December 19 It's been a long, exhausting, frustrating and frighteningly expensive few weeks, but we've finally managed to overcome the load on the servers. By the looks of the board a lot of you are wondering what took so long...here's the condensed version of what's been happening.. About 4 weeks ago Bang Bus's bandwidth went from 50-60Mbps to 100Mbps. We were already adding servers so we put the new machines online and the traffic immediately jumped to 200 Mbps. At that point we started upgrading all our hardware to handle 400 Mbps..figuring that doubling what we were using would be sufficent. By the time the new servers arrived we were already doing 350Mbps. The minute the new servers went online it jumped to 630Mbps and eventually peaked at 680Mbps. In case you're wondering just how much that is...none of the hosts we've spoken to (Exodus, Level3, Global Crossing, XO and most of the other major providers) have seen anything like this. We'd been anticipating growth and were acting accordingly, but you really can't anticipate something like this. As it stands now we have capacity for about 1Gbps and our architecture will scale up to 2Gbps by simply plugging more servers into the clusters. Static content is served by a cluster of nine machines each capable of pushing 100Mbps. The cluster should withstand failures on up to 3 machines simultaneously without noticeable slowdowns. Dynamic content is served by a 3 node cluster with each machine capable of handling the entire load if the other two were to fail for any reason. The database machine is currently the only single point of failure so it's the most powerful machine of the bunch and has built in hot swappable spare hard disks, network adapters, power supplies, and RAID controller. To try and overcome the single point of failure I've been rewriting most of the critical code to try to make everything at least continue to function during database failures. I'll also be adding a dedicated realtime backup server tomorrow that can be temporarily switched on in the event of problems. All the machines are connected to a high end load balancing switch that scales up to 2Gbps. We've got spare parts and machines on hand at Candid in case of problems. We're working with a company that specializes in streaming video to build a scalable solution to do the Windows Media streaming...the fast forward/rewind/seek functionality. ETA is early January on that...until then the video will continue to stream from the web servers. All planned upgrade work is now done. I'm sure there are still some oddities left to deal with, but for the most part everything should be working and pretty damn fast. Normal update schedules will be resuming Thursday. I know the last few weeks have been trying for everyone, but you'll just have to trust me when I say it's been ten times as bad for us;-) Try staying up 48 hours to bring $50k worth of hardware online only to find out the minute you go live your bandwidth jumped another 300Mbps and your sites are still barely crawling along. We're pushing things to a whole new level and it's tough to foresee every obstacle that will arise along the way. Now for the processors... They're just as fucked up on our end as they are for you guys. We deal with them constantly throughout the day trying to get to the bottom of things and get your problems resolved, but they hold all the cards so when we say "Jump" they say "Go fuck yourself with a rusty spatula". Not in so many words of course, but you get the idea. I'd been planning to be pretty close to completion on our own billing software, but the server nightmare has taken up all my time this month. I'll be picking up on it after Christmas and would say the tentative ETA is mid-February. I'd like to begin beta testing it by mid-January so a few of you braver souls can jump in on that if you just can't stand iBill/CCBill any longer. That's pretty much it for the moment... I've got ~5000 emails to sort through over the next couple of days so if you've got something urgent call the office during the day or my cell (305-562-0357) after hours.