Your x2k.com sites are down

I don't think this is a "DNS outage" as people are reporting they can't even ping their boxes.

Why would National Net say it was a DNS outage?

That is a fair question....the original diagnosis was DNS related but in actuality there was a deeper issue in our routing that it took some time to diagnose and then isolate. In an effort to get information out to our customers we probably spoke to soon but it was all in an effort to keep you guys posted. Again, sorry for the issues and I assure you that we will have this issue resolved soon.

--T

Working now for me :)

Sounds like a routing table issue. :o(

looks like its resolved, thanks Tony and the natnet techie team for your good work!!!

Hi Brad...as people that know me can attest, I am not the technical brains of the company so I honestly do not know...what I can tell you is that once the dust has settled and everything is back up and running flawlessly we will go through an entire Post Mortem process that will detail every tiny thing that could have caused it, we will analyze how we diagnosed it, and we will define new ways of both avoiding the issue in the future and how we handle future issues.

As I have always promised my clients...we cannot guarantee you perfection...nobody in this business can....but what we can guarantee you is that when troubles do arise we will be mobilized and working on it with ever resource at our disposal.

Again, I am deeply sorry for this issue and I truly appreciate your patience.

--T

Thank you Tony!

I agree, its more than just DNS servers at this point... I was able to ping the boxes earlier.

Down again :(

I did not see that?? but thanks 51 times ;o)

Yep... down for me again, too.

All is back up and well. Thanks T!

Please let us know what the issue was when you figure it out.

Are you guys sure you reset the router?

http://www.techienation.com/wp-conte...giantdlink.jpg

still up and down, this is costing a small fortune in lost sales, really hope you guys are close to a fix

It's not just you. A lot of people now consider spacash to be as good as dead.

Going on 4 hours of downtime now. This REALLY hurts. :(

natnet online checker, click here:
http://www.shemp.com

Thanks Tony !!!!:thumbsup:thumbsup

I take that back. ugh. going on 5 hours being down :Oh crap

Hey Guys....

I totally understand your frustrations....believe me, we are right there with you. I assure you we are working with everything we have to get this resolved. We have the problem isolated but have not yet gotten it totally under control. We have brought in a team of engineers from Cisco that are working with our team and will remain engaged through the resolution. I am hoping that it will not be much longer....just bear with us.

I am so very sorry for the problems....we have been doing this for 15 years now with many, many years of great service but it still makes it no easier when just one customer is having an issue let alone the whole bunch. Thanks so very much for your patience.

--T

Looks like they lost everything after their edge routers @ 56Marietta, probably a fiber transport issue. Our peering with them is still online with them @ 56 and they are fully announcing routes.

It's alright. It happens... but goddamn: Who else can take down half of the adult industry?

Thank you for keeping us all informed.

http://gifs.gifbin.com/052011/130461...leweed-gif.gif

:)

Thanks for doing your best to get the sites back up asap and for taking the time to keep us all informed about the progress. :2 cents:

Seems like it's back up now,**** F.U.B.A.R. Webmasters****, Proudly Hosted On NATIONAL NET:thumbsup:thumbsup

http://www.fubarwebmasters.com/index...0e5e4d583 52e

Shit is happening.

They are indeed back up - just received my invoice lol...

:)

You are a class act Tony. I hope you are well, my friend.

Working fine.

They've been up since noonish PST here. NatNet are not slouches! They are always on the ball. I think today was the most downtime we've had in the many years we've been with them and it wasn't that much.

Good work guys.

Anyone know what actually happened?

Best gif ever. :)

No, i thought i might of woken up to an email explaining it all, but nothing.

Right. There was talk about post mortem explanation, it would be interesting to hear why thousands in profit went down the proverbial shitter. :disgust

Hey Guys....

To answer your question, over the next several days our engineers, along with engineers from our core hardware providers (Cisco, Force10, Dell, etc), will be conducting an investigation that will result in a detailed Post Mortem. In fact, as I type this there are Senior Level engineers from our providers from California on a plane to Atlanta that will be working with our team to conduct the investigation. The PM will detail what went wrong, the steps we took to correct it, what steps that we could have taken to expedite the resolution, and how we can prevent this event from happening in the future. Not only will we use this investigation as a way for us to becomes a better provider and our vendors will use it to aid in helping their customers avoid the same issue in the future. Going through an outage is the worst possible feeling for a hosting company, but it is our belief that if we are going to go through it we may as well learn something from it.

As many of you have heard me say over the past 15 years, I cannot promise you perfection....no hosting company can....what I can promise you is that when something does happen we will be here using every resource at our disposal to get you back up and running and that is what happened yesterday. It has been several years since we have had any type of network wide outage so my goal is to make it a bunch more before we have another one (crosing fingers) ;)

As always, if any of our customers has any questions, comments, or concerns they can always submit a ticket within MyNatNet or call our NOC 24x7x365 at 1-888-4-NATNET. We are always here for you!!

Once again, I am so very sorry to all of our clients for this temporary outage. Please trust that we feel your pain and share in your frustration when anything goes wrong and we will give everything we have to insure that it never happens again. That, I promise.

--T

YOU are the man!


:thumbsup

Sounds good, interested to hear the outcome of the investigation. :thumbsup

I've never had more than 10 minutes of downtime at Isprime in 10 years. just saying.