NATNET Clients and Friends...Outage Report
 

Hey Guys....

Just a quick note to let you know that we are aware of the issues and are working to resolve them now.

All that we know at this point is that simultaneoulsy something affected our entire HVAC infrastructure. This caused the temperature to rise in our Data Center and is causing heat loads on several peices of equipment. This is why some servers are affected and others are not.

We have HVAC people on site working on the situation, and we have called in for portable Air Conditioning that should be setup within an hour or so in case the normal systems are not repaired by then. Either way, we dont anticipate the sporradic outages that you are seeing to last much longer.

Thanks for your continued trust and confidence, and rest assured we will get this completed ASAP.


Tony Morgan
President
NationalNet, Inc.

you guys rock Tony :)

p.s... buy my sig ;)

Outages can happen to anyone, sorry to hear that T.

I'm sure they'll get things up soon.

Professional.

As any good host, address the issue on GFY within time.

THAt speaks further than a fucking server special.

Ill put my money with Nat Net or Webair because of service like this.

excellent support.

I didnt see my NatNet tour go down but others did see downtime, awesome to know its all good and bumping.

cheers all.



:glugglug

yup, that's why no one can claim 100% uptime... these things happen, completely beyond anyone's control.

Good of you to take the time to come and make an announcement on the situation yourself Tony, very admirable of you :thumbsup

the servers that are down, are overheating ?

Nice to see the Boss up at 03:00 :thumbsup

Thanks for the kind words everbody.

Yes, this type of thing can happen to anyone. It is why we always hate hearing about any hosting company having issues because we know it could just as easily be us.

Brujah....it is not exactly that the servers are overheating...it is the electrical systems that are suffering the effects of the heat. UPS's and Power Strips do not work well in the heat, so they kick before anything dramatic would ever affect the server. I hope that answers your question.

Shemp....Well, I am sure that anyone in my position would do the same thing. We actually have a very good "Crisis Team" made up of 3rd Level System Administrators, Network Engineers, and Senior Management that can be assembled in less than 30 minutes for occassions such as this. Of course, tonight I would rather have a team of Air Conditioning mechanics on staff!! LOL!

We are working to power down all non-essential, non-customer equipment that is running in order to bring back up more of the sites that are down but we will not be able to be back in full production until the Portable Air arrives or until the HVAC Infrastructure is repaired.

I will keep you posted.

--T

open some windows

ill fedex a big fan for my server

On a serious note, I have all confidence in the nat net people..

:thumbsup

thnx for letting us know what's up Tony... I'm sure you guys will get it solved ASAP.

Seems to be a bad week for adult hosting companies. NatNet is top drawer though, so if you have a server there you wont be down a second longer then absolutely neccesary.

A quick update....

The first set of HVAC units are coming back online as I type this and the redundant systems should be online soon after. The portable units are being installed anyway, as a fail safe, until we can determine what happened and put together a plan so that this is never repeated.

Once the HVAC system and the portable units are all running it will take a little while for the Data Center to cool back down to optimal temperature so that we can begin to reboot all of the devices that are still down.

The good news is that we are on the downhill side of this now....thanks for your continued trust and support!


--T

i pity the poor techs that were on duty when this happened.

Thanks for letting us know Tony, very professional....

For how many hours has this been going on?

I think in my close to 2 years at nat net..this is my first downtime I am aware of.

Almost there....

All of the HVAC systems are up and operational and the temp is dropping faster than expected. We hope to be able to have everything back online in just a little while.

I will update again when we are done.


--T

OK....last update....we are at the finish line :)


All of the repairs to the HVAC Infrastructure are now completed and the systems are running in normal mode. However, the heat caused us to lose one phase of power throughout our data center....it is the equivalent of "tripping a breaker" in your home. In order to resolve this we are going to be forced to power cycle the entire center. This will take place in the next little while and will create a very short outage (somewhere between 2 and 10 minutes). Once everything comes back online this entire experience will be behind us.

If you are a NatNet client and have any issues after the power cycle is complete, please contact us through http://www.mynatnet.com or by calling the Data Center toll free at 1.877.471.9075 Option 1.

Rest assured that we are going to learn from this and we have already put measures in place to be certain that this is never an issue again for our valued clients.

As always, thanks for your continued trust and business.


--T

Good job...Very professional..Good luck with your business

Andre

calling now thanks

Thanks For the update T.......its communication like this at a time of crisis that makes you guys Tops:thumbsup :thumbsup

My server is still down

everything's up for me again, good work Tony :thumbsup

Can't you read sweet's post above??

tony and natnet would be my 2nd choice for hosting. topnotch guys and service.
:thumbsup

There could have always been a secondary system to cool the place down incase something like this should happen :)

i thought redundancy would have been mandatory considering the volume running through

ATL without AC - I am suprised anything is working!

NAT NET has always been top notch, something's are beyond anyone's control!

good to see everything is being resolved... great response Tony...

I"m going to use this thread to copy and paste to the webmasters who were whining about stats being down.... b/c they were afraid they might miss their one sale a month ;)

pick up the phone and give them a ring for fastest service.