Major down time

[theking]

I received this from my host at 2:32 PM Pacific Standard Time.

We regret to inform you that you may be affected by a recent failure of one of our switches. If your site is currently unavailable, then you are on the segment of our network that is unreachable at the present time.

We are working to replace the faulty piece of equipment and should have service restored within the next few hours.

We apologize for this inconveience and assure you that steps are being taken to limit the future possibility of this type of outage.

Question...Is this something that they should have had a backup in place? What would be a normal time frame to correct this problem?

It is now 7:13 PM and my site is still down. About a week ago I had 12 hours of down time because of some kind of failure. It is a main stream site and I am still awaiting my business license (reminds me that I need to call Monday and see why the delay), so I am not losing much if any money, but may be losing some traffic that I was just beginning to build and I did lose an affiliate program that I had applied for.

[.:Frog:.]

What host?

[pr0]

If they don't see this post & offer you 2-months free hosting....move to another host.

And no i refuse to spam any host in this thread, but if they don't make it right...move somewhere, anywhere & tell them to fuck off.

[theking]

Originally posted by .:Frog:.
What host?

As I stated when this happened the last time, I do not care to identify the host at this point in time. They may read this board and I don't want to disturb whatever relation that I may, or may not have, with their tech support. If and when I decide to change hosts then I will identify them as being a host to avoid. Also as I explained before the host is one that my dad had signed up with and as near as I can tell (from going over his old E-mail and reading trouble tickets) he switched to them about a year ago and though there has been some problems with them it was not really down time issues and the problems that he did have seem to have been resolved in a reasonable time frame. Maybe they are just having a bad hair week. If it was costing me alot of lost revenue I would not be as lenient in my thinking.

[Jamie]

If it's just a switch that went down, the only down time you should see is them unplugging your servers ethernet from the old switch to the new switch.

ofcourse if they dont have a backup switch then they are most likely fucked.

switches are by nature dumb, so replacing one should honestly only take a few minutes. They don't require too much work to get them configured and working

[frankfortuna]

A good provider should have a backup of the config and a spare switch available. It sounds as if they either have a greater issue such as lack of equipment, expertise, or a non-premium service agreement with their hardware provider.

In a perfect world, one who is properly equipped should be able to respond and correct this type of issue in under a couple of hours.

[theking]

Jamie and frankfortuna

"We apologize for this inconveience and assure you that steps are being taken to limit the future possibility of this type of outage."

It appears that they did not have the hardware immediately available but I got the E-mail at 2:32 PM so could they have not acquired another piece of hardware within this time frame? It is now 8:15 PM. What is the cost of this piece of hardware?

[Cirrus]

verio?

[frankfortuna]

Depending on the type of switch, the costs and availability could vary wildly. In some areas, you can't simply walk into a store and purchase a blade for your larger Cisco Catalyst chassis.

The greater question is, why don't they have on-site maintenance? Particularly if they are a major provider, which I'm assuming they are.

[<IMX>]

theking--
This is just IMHO, I will never host with a mainstream host again. You have no idea what they are doing to those servers, and they do not know the meaning of customer service.

At least with adult hosts you can immediately go to half a dozen webmaster boards and bitch about them. If they are large, they will generally be accountable to a literal forum of their peers even if your individual account is small.

[letshunt]

A switch? Bullshit....it's either a router or part of a D-Slam they rigged to route traffic. A switch is a dumb box....it just relays whatever the packets are earmarked for...with the exception of programmmable switches.

A router is a little different deal....big trouble there, but most backbones have redudancy in router configs.

[theking]

Originally posted by letshunt
A switch? Bullshit....it's either a router or part of a D-Slam they rigged to route traffic. A switch is a dumb box....it just relays whatever the packets are earmarked for...with the exception of programmmable switches.

A router is a little different deal....big trouble there, but most backbones have redudancy in router configs.

My main concern and question is, so that I may understand what kind of host I am dealing with, is this a problem that they should have had back up in place? Do the best of hosts have reduncancy so that if a piece of hardware fails the down time would be minimal say and hour or so? Or can the unexpected happen that no host is really has redundancy in place to contend with the problem?

[Honeyslut]

m3server.com

[theking]

Now this on their main page.

"Network Status: We are currently in experiencing an outage on our 157.xx.xx.xx segment of our network, This outage is hardware related and should be resolved no later than 2pm PST. (11/16/02 - 08:40 AM PST)"

WTF, did the terrorists bomb their facility????

If in fact it takes until 2:00 PM PST that will mean 23 1/2 hours of down time. I wonder how many customers this is going to cost them?

[Honeyslut]

Originally posted by Honeyslut
m3server.com

Cough

No down time at m3server

[theking]

Originally posted by Honeyslut

Cough

No down time at m3server

I saw it the first time. I assume that you are giving them a recommendation and if so based upon personal knowledge? By the way it is 12:30 PST and still nada. I have doubts that they will make the 2:00 PM PST deadline.

[theking]

Just received this.

"We expect to have the 157.xxx.xxx.xxx segment of our network back online within a few hours. Servers will come back online sequentially.

If there are any changes needed on your behalf, you will receive an email indicating such. The IP address of your website(s) will change as a result of thie restoration. You will receive further instructions as they are needed.

We appreciate your patience during this unfortunate network outage.

All customers affected by this outage will receive account credits as a result of this 24 hour period of downtime.

Thank you again,"

My server is still down and it says within a "few hours". I wonder what a few hours will be?

[Honeyslut]

Originally posted by theking


I saw it the first time. I assume that you are giving them a recommendation and if so based upon personal knowledge? By the way it is 12:30 PST and still nada. I have doubts that they will make the 2:00 PM PST deadline.

Yes, I have all my sites on a server there. No problems

[Wenchy]

I'm with Honeyslut... I host at M3Server as well, and have never had any downtime.

Get in touch with dantheman... he'll take care of you

[theking]

Well it is 29 hours and counting.

"If there are any changes needed on your behalf, you will receive an email indicating such. The IP address of your website(s) will change as a result of thie restoration. You will receive further instructions as they are needed."

I have been thinking about the above quote. Does the changing of the IP address mean that my domain will have to propagate through out the World Wide Web again? If so, this means I could be down for another 24-72 hours. Any opinion about this?

[theking]

Now this is on their main page.

"Network Status: We are finalizing the installation of a replacement router/switch that will restore service by 12:00am EST."

[theking]

Originally posted by letshunt
A switch? Bullshit....it's either a router or part of a D-Slam they rigged to route traffic. A switch is a dumb box....it just relays whatever the packets are earmarked for...with the exception of programmmable switches.

A router is a little different deal....big trouble there, but most backbones have redudancy in router configs.

It appears that you hit the nail on the head. Am I to assume that they did not have redundancy?

[Deranged Monkey]

damn, 29 hours? I know what I would be doing...

[.:Frog:.]

switch hosts maybe?
I would.

[theking]

Originally posted by .:Frog:.
switch hosts maybe?
I would.

I intend to E-mail them and let them know that if this ever happens again they will lose me as a customer. They have claimed that they are taking measures so that it doesn't happen again. Over the past six years I have had experience with a dozen or more hosts and there have been problems with each including down time. There really is not a perfect host, no matter how much you pay, there are certainly some some better than others.

[theking]

Well fuck. It was 2:00 PM PST then 12:00 AM EST and still nada. 30 and 1/2 hours and counting.

[Honeyslut]

Talk to dantheman !

[theking]

Well, I am about ready to give up for the night. It is almost 34 hours and counting. Being this late I doubt if it will come on line tonight.

[L0stMind]

there is no way in hell you should be done for this long due to a dead switch. that is a crock. What hosting company cannot afford to have a SINGLE spare switch in place? Each switch I own has a spare waiting just in case - it's not prohibitively expensive.

I bet your host had a problem with their upstream provider or something similar and are covering up.

[funkmaster]

"Well, I am about ready to give up for the night. It is almost 34 hours and counting. Being this late I doubt if it will come on line tonight."

... and in your first post it reads

"so I am not losing much if any money, but may be losing some traffic that I was just beginning to build and I did lose an affiliate program that I had applied for."

... you really look after your buiz ... wonder how you would react if there was traffic on the site ... a very hight volume of traffic ... like ... 10 clicks per day ... or maybe 15 ...

[theking]

Now this is has been posted on their main page.

"Network Status: Upgraded router has been successfully installed and our staff is in the process of changing site IP addresses to coincide with the newly added premium bandwidth connection. Please check back for updates."

My site is still down. 40 and 1/2 hours and counting.

[dantheman]

I've been watching this thread, thank you honeyslut and wenchy for the kind words

theking, sorry I cant help you much with the current problem( I do hope they work out their problems for you and all thier clients.)

But if I can do anything let me know.
dannyh at simplecom.net.

[theking]

Originally posted by dantheman
I've been watching this thread, thank you honeyslut and wenchy for the kind words

theking, sorry I cant help you much with the current problem( I do hope they work out their problems for you and all thier clients.)

But if I can do anything let me know.
dannyh at simplecom.net.

I have a question for you. From what sparse information that has been provided to me by the host, that I have posted in this thread, do you have an opinion as to the real problem, and should have redundancy been in place to avoid the problem?

[dantheman]

it's really hard to tell, I can only go by our setup, downtime will always be a issue when theres so much networking taking place and of course the internet is one big ass network,
BUt you have to forsee some things and act accordly(sp) 30hours is MAJOR downtime, We've never had anything like that. yes of course we have some downtime but you have to have reduncies built in to avoid as much as possible.
Sounds like they had a major router go down and had to order another one to bring thier network back online, and when you have to order a piece of hardware you have to have it shipped/bought in to make the change.

Please remember all this is is speculations on my part.

please ignore the misspellings and whatnot, I'm a little hungover and glased right now.

[theking]

Back on line but loading slow and cannot access my admin panel or FTP.

[theking]

Here is their explanation as to what went wrong.

"Here is a manifest of the problem thay occurred over the weekend. Normall, this is not made public, but we thought it would help explain the outage in greater detail.

Friday, November 11, 2002 - 1pm CST - Monitoring system indicates that Cisco switch has failed. Vendor called for 4 hour emergency replacement. REplacement arrives, replacement is not functional. Vendor chooses to run through diagnostics to get the replacement to work, rather than replace the replacement. Total time elapsed = 14 hours. Vendor orders replacement after failure to get initial replacement to function properly.

Saturday, November 16, 2002 - Second replacement switch arrives, must be installed and configured on new network IP range. Re-IP process begins on dedicated and shared servers at approimately 11pm CST.

Sunday, November 17, 2002 - Re-IP process completed at approximately 7pm CST. All sites/servers appear to be functioning properly.

Once again, thank you for your patience and understanding."

[Auslander]

The King,

Anyone that buys from a hosting company that doesn't implement redundant routers, switches, and/or backbone providers, etc. is asking for trouble in the first place.

I understand 100% about getting faulty equipment, etc. however, they should have had fail-over switches implemented.

Bad...bad...bad...I would move...and move quickly...