National Net in down?

[design]

Anybody expiriencing down-time right now?

[=^..^=]

Makes sense
I just tried to load a site I know is on nat net and it was down

[design]

I can't even access support url and their main site in down.

[Trax]

yep
same here
looks as if thats the case!

[eadweb]

Yup, It's been down for almost 45 minutes now

[IanUK]

Seems like its just one thing after another with them lately

I wish Mach10 were still here...

[Roald]

maybe they also stole their servers

[SpeakEasy]

Yup, been down here for about an hour or so.

[Jon2]

that one records site wasn't loading for me...prolly hosted on them too...

If natnet is down...Doesn't that mean the world is coming to the end? I always thought they were one of the most trusted and never down hosts out there?

Jon

[SweetT]

Hi Guys....

Yes. I just got the wake up call saying that unfortunately all of NationalNet is currently down. Our Emergency Response Team is assembling in the Data Center now to find out what is wrong and to get it corrected as quickly as humanly possible. Regretably, that is all I currently know.

I will keep you posted as I know more.

I am truly sorry for this interruption in service and I can assure you that we will make it right by our clients.


--T

[Mike-BP]

lots of downtime lately at natnet

[Warden]

Well this thread answered my question

[BaldBastard]

" lots of downtime lately at natnet "
They were down for 30 minutes last week is all I noticed, its so regular that the first thing I did was check out CNN as I figured 1/2 the US must of been up in smoke.

[ReneK]

my server is also down.... no update yet ?

[emmanuelle]

maybe their tech guys are all out at bus driving school

[12clicks]

Originally posted by Mike-BP
lots of downtime lately at natnet

bullshit.

[Mike-BP]

Originally posted by 12clicks
bullshit.

it's already the second or third time this week

[Mason0]

Originally posted by 12clicks
bullshit.

Bullshit to your bullshit. They HAVE been down a lot lately.

[SweetT]

An update and more information for you:


First, when someone says there has been alot of downtime lately they are right. A single instance of downtime is alot to us. However, what you must understand is that when you grow to be as big as NatNet you can't even have a "minor blip" without it making Headline News. So, a couple of weeks ago when we had an issue with a few high profile servers, it seemed like we were having huge issues...when it was very isolated. This coupled with the issues of today make it seem worse than it is....however, right now none of us are happy, right?

Now, the update.....

Our ERT has isolated the issue and is working to repair everything and get it back up and running at peak performance. I don't have all of the details yet, but what I do know is that it is related to Power and that Power to the entire Data Center has been brought down in order to make the necessary repairs.

As soon as the repairs are made and the Data Center begins to come back up I will be back with an update.

Thanks for your continued trust and confidence.


--T

[Rick Latona]

I'm across the street from the datacenter. I don't see how it could be a back bone issue since I'm online.

[Rick Latona]

Originally posted by SweetT
An update and more information for you:


First, when someone says there has been alot of downtime lately they are right. A single instance of downtime is alot to us. However, what you must understand is that when you grow to be as big as NatNet you can't even have a "minor blip" without it making Headline News. So, a couple of weeks ago when we had an issue with a few high profile servers, it seemed like we were having huge issues...when it was very isolated. This coupled with the issues of today make it seem worse than it is....however, right now none of us are happy, right?

Now, the update.....

Our ERT has isolated the issue and is working to repair everything and get it back up and running at peak performance. I don't have all of the details yet, but what I do know is that it is related to Power and that Power to the entire Data Center has been brought down in order to make the necessary repairs.

As soon as the repairs are made and the Data Center begins to come back up I will be back with an update.

Thanks for your continued trust and confidence.


--T

Thanks for the update.

[kenny]

yes has been down for me since I first tried a hour and a half ago

[Phoenix]

Thanks for the update Tony

[SpeakEasy]

hum, No back up power?? humm

[Phoenix]

Originally posted by SpeakEasy
hum, No back up power?? humm

i am sure they have backup power systems.

Tony, i sat next to a power supply guy from up here..they sell to the four 9's and five 9's

if you guys have to upgrade, i can give you his contact info

[SweetT]

Originally posted by SpeakEasy
hum, No back up power?? humm

Even though nothing seems funny right now, it is almost funny that you mention back up power because at this point it seems that is what may have caused our problems.

I am still fuzzy on some of the details and I really probably shouldnt post until I have more concrete answers. I really want my staff to be fixing the problems rather than giving me updates, but from what I can peice together here is what has happened:


Last night we did some routine maintenance to our power systems in all three of our Data Centers located here in Atlanta. At the conclusion of the maintenance window we placed our power systems back into the Primary System with the Redundant System at the ready. Over the course of about 5 minutes the system Automatically switched back and forth between Primary and Redundant about 10 times for reasons we do not yet know. This switching back and forth has caused some sort of surge through the system. This has apparently blown the three major "fuses" in each Data Center. The fuses are what sit at the end of each phase of our three-phase system. This takes out both the primary and redundant power supplies.

Currently we have all three Data Centers dark while we replace these fuses, but these are not like the fuses that you have at home....these are huge industrial type fuses that take time to replace and must be done in a very methodical way. As soon as this is completed we will begin bringing the three Data Centers back up just as methodically.

Hope this helps a little bit. I will be back with more updates as I know them.


--T

[b00bies]

I see some natnet sites being back up

cjbucks, teeniesxxx, etc

but mine arn't yet hehehe but hopefully soon!

[bluff]

Originally posted by b00bies
I see some natnet sites being back up

cjbucks, teeniesxxx, etc

but mine arn't yet hehehe but hopefully soon!

not up yet

[JFK]

Originally posted by SweetT
Even though nothing seems funny right now, it is almost funny that you mention back up power because at this point it seems that is what may have caused our problems.

I am still fuzzy on some of the details and I really probably shouldnt post until I have more concrete answers. I really want my staff to be fixing the problems rather than giving me updates, but from what I can peice together here is what has happened:


Last night we did some routine maintenance to our power systems in all three of our Data Centers located here in Atlanta. At the conclusion of the maintenance window we placed our power systems back into the Primary System with the Redundant System at the ready. Over the course of about 5 minutes the system Automatically switched back and forth between Primary and Redundant about 10 times for reasons we do not yet know. This switching back and forth has caused some sort of surge through the system. This has apparently blown the three major "fuses" in each Data Center. The fuses are what sit at the end of each phase of our three-phase system. This takes out both the primary and redundant power supplies.

Currently we have all three Data Centers dark while we replace these fuses, but these are not like the fuses that you have at home....these are huge industrial type fuses that take time to replace and must be done in a very methodical way. As soon as this is completed we will begin bringing the three Data Centers back up just as methodically.

Hope this helps a little bit. I will be back with more updates as I know them.


--T

Thanks for the Update T .... I see you havent blown your Fuse yet, so everything is OK

[xxxdesign-net]

WHos hosting "how I got rich " sites like http://crackwhoreconfessions.com ?? ...been down for a day I think...

[Buddy]

I'm back up.
Thank you for keeping us posted.

[Furious_Male]

Stats Remote is back up as well. Looks like they are slowly coming back.

[[illnet]-Jamie]

We are half up and down
Spoke to bill and they are working on it ..turning racks on one by one

Slow process

[Tofu]

Im up...FA's everywhere are happy once again.

[SweetT]

Another Update:


As you can see things are coming back up but this is a very slow and very methodical process. Currently we are at about 75% and we should be getting closer to 90% within the hour. As with any full power outage there will be a few machines that will be tricky to get back up and these will take priority once we reach the 90% mark and they should be up shortly thereafter.

As always, thanks for your trust and confidence and I will continue to keep you posted.


--T

[SykkBoy]

Thanks for the updates!

This is what customer support is all about, keeping your customers informed.

While there is no such thing as 100% uptime (as much as we'd love it in a utopian webmaster society) it's refreshing to know that some companies really care about their customers and keep them informed and we know the gang at NatNet will make it right.

[50 Cent]

they seem to have 95% of their servers back up

[b00bies]

why must i be the other 5%

is there someone i can contact to get mine booted up

[IanUK]

Mine is also still down, I also raised a looong ticket in support about 30 mins ago (it was back up) and when I clicked to submit it I got dns failure again - pissed off is not the word !!

These recent outages have really taken the shine of Natnet for me, Mach10 never had these problems.

I do appreciate the responses here from their staff, but i'd really rather these outages didn't keep happening in the first place.

PS. are ex mach10 customers deliberately switched back on last or am I just paranoid ;)

[SteveLightspeed]

Originally posted by IanUK
PS. are ex mach10 customers deliberately switched back on last or am I just paranoid ;)

No, you are just paranoid. I've been with Natnet forever, and we are still down too (all our tours are offiline)

PISSED OFF DOESN'T QUITE FIT MY MOOD RIGHT NOW.

Livid is a better word.

Steve Lightspeed

[optional]

Originally posted by IanUK
PS. are ex mach10 customers deliberately switched back on last or am I just paranoid ;)

paronoid.. I'm ex mach10 and have been back up for a while ;-)

[IanUK]

I am sitting with my tin foil hat on.....

I just cheered myself up with a laugh at the following from NatNets Tour on their site (which is up btw):

------

Multi-redundant power system

Without power, nothing works! Therefore, NationalNet has gone to great lengths to back up all power systems. Present in the data centers are battery systems capable of running all the equipment for over two hours.

In the event of a power outage, NationalNet has two separate power connections to generators on the top floor and basement of the buildings. The generators start up within ten seconds of an outage if the power has not been restored.

--------

Now doesn't that make it all seem better ?

Ian
(whose site is still down)

[pussyluver]

Originally posted by Lightspeed
No, you are just paranoid. I've been with Natnet forever, and we are still down too (all our tours are offiline)

PISSED OFF DOESN'T QUITE FIT MY MOOD RIGHT NOW.

Livid is a better word.

Steve Lightspeed

Sounds like your vendor is doing his best. They had UPS etc. So should you still be pissed ???? From the outside looking in, its a hard call. Shit happens no matter how hard you try.

You put all your hosting eggs in one location. A truly paranoid person would have servers on different sides of the country. I know Lightspeed is a major player of course. I don't know if the volume of business warrants fall backs like some sites do.

I do know there is a down side to all this. With thousnads of galleries in my TGPs, I autoscan for broken links. No big news there, many of us do. Point is that could hurt Lightspeed if a few TGPs deleted hosted galleries. The the lost biz on tours being down andpissed users that might drop their accounts will add up.

This turns out to be a hard thing to manage for all of us. I've decided to live with the broken links for today. That might be harder for a shemp or sleazy to do??

So what will you do, Steve? Beat on your hosting provider? Add redundant servers? Switch providers? A fluke like this could happen to any of them I suspect? They have been very open and didn't just point a finger at someone else.... At the dollar level, that might not count. Interested in your thoughts on a course of action?

Wonder if there is loss business insurance for the adult world for situations like this. I know my mainstream biz has it.

[Theo]

All hosts will go down once in a while. There's no such thing as 100% uptime and if you had a server for more than a while with 0% downtime, you should consider yourself lucky because not all the company's clients will be in position to claim the same. Having Akamai going down for hours is enough proof.

[pussyluver]

Originally posted by Soul_Rebel
All hosts will go down once in a while. There's no such thing as 100% uptime and if you had a server for more than a while with 0% downtime, you should consider yourself lucky because not all the company's clients will be in position to claim the same. Having Akamai going down for hours is enough proof.

Talk about a quick response. Got to give the engineers credit on this one. Within a couple of hours most big names were up again


For what it is worth I checked at 9:00 CDT and there were three root servers down in the US. Those three are still down at 2:00 pm.

[b00bies]

seems lightspeeds back up... most big names are back up

and anonymous old me is still down

[SteveLightspeed]

Originally posted by pussyluver
Sounds like your vendor is doing his best. They had UPS etc. So should you still be pissed ???? From the outside looking in, its a hard call. Shit happens no matter how hard you try.

You put all your hosting eggs in one location. A truly paranoid person would have servers on different sides of the country. I know Lightspeed is a major player of course. I don't know if the volume of business warrants fall backs like some sites do.

I do know there is a down side to all this. With thousnads of galleries in my TGPs, I autoscan for broken links. No big news there, many of us do. Point is that could hurt Lightspeed if a few TGPs deleted hosted galleries. The the lost biz on tours being down andpissed users that might drop their accounts will add up.

This turns out to be a hard thing to manage for all of us. I've decided to live with the broken links for today. That might be harder for a shemp or sleazy to do??

So what will you do, Steve? Beat on your hosting provider? Add redundant servers? Switch providers? A fluke like this could happen to any of them I suspect? They have been very open and didn't just point a finger at someone else.... At the dollar level, that might not count. Interested in your thoughts on a course of action?

Wonder if there is loss business insurance for the adult world for situations like this. I know my mainstream biz has it.

Well, I'm sure Natnet will make it right. I am pissed about lost sales. But I can't think of any company I'd rather have on my side in a technical firefight than Natnet. What may be 6 hours of downtime for them would be 12 in most other places.

Currently, some of our servers are up, some are down. We are taking affiliate traffic, but since our tour server is down, its going nowhere, and there are no sales. So when your Lightspeedcash ratios are 1:5000 for the day, hopefully you won't post on every board how much Lightspeedcash sucks. That is my biggest worry right now.

Steve Lightspeed

[[illnet]-Jamie]

Originally posted by SweetT
Another Update:


As you can see things are coming back up but this is a very slow and very methodical process. Currently we are at about 75% and we should be getting closer to 90% within the hour. As with any full power outage there will be a few machines that will be tricky to get back up and these will take priority once we reach the 90% mark and they should be up shortly thereafter.

As always, thanks for your trust and confidence and I will continue to keep you posted.


--T

Any idea when the rest of my boxs will be back up?

I know your on it and alll.....but any time frame to ease myself...


Thanks T

[TheDoc]

We are back up... finally ://

[AmeliaG]

Damn it, I was just fixing to switch my servers to National Net