Here is another update.....
First, I want to thank all of you for keeping your head up during this trying time. An all-out outage is something that hasn't happened to us in a very, very long time and it is nice to see that people don't forget the good times when we do have the occasional stumble.
Now for the good news.....
All of the power has been restored to every rack in all three Data Centers. As happens in cases like this we are working on a few stubborn servers to come back up. This is not due to lack of power but due to the hard crash that they experienced when the original power was lost. This is common in situations like this and should not take too long to completely rectify. We have our entire staff of System Administrators and Network Engineers working on these few machines so you can rest assured that we will have them back online post haste.
We will be working throughout the night to learn exactly what caused the issues and what can be done to prevent them from happening again. As we do with any outage, large or small, we will be issuing a detailed, written post-mortem explaining everything....but this time we are going to post this to
http://www.mynatnet.com for all of our customers to see. This should be posted by tomorrow afternoon at the latest and we will leave it up for at least 24 hours to give everyone a chance to see it.
Again....and I can't stress this enough.....THANKS for your continued trust and confidence even through all of the trials of today. If there is a highlight of this day is to see some of our largest clients recommending NationalNet even when we were in the middle of the longest outage in our Companies history. It really puts it all into perspective.
Thanks guys!!
--T