National Net down?
 

Can't access my sites or nationalnet.com to put in a ticket

:helpme shitty man

Hmmm this has happened twice in the last month!

my sites seem fine.

:Oh crap

Some of ours are up, some are down

Oh well, everyone is going to have downtime at some point, its inevitable

If they are down, I can 100% guarantee they know about it and are working on the problem.

As an aside, am I the only one who has tech support numbers written (typed) down in case there's a problem with the interwebs??!!

thank you... :thumbsup

Hey All....

I am sorry to admit that, yes, there is something going on in our network. The guys are all over it and it will be corrected soon. From what I understand it is only affecting parts of our network but we will not stop until the network is at 100% again.

I will let you know when the issue has been corrected.

Again, I apologize for the inconvenience but I can assure you that our staff is on top of it and it will be corrected as soon as humanly possible.


--T

That is the beauty of Natnet, they are normally all over fixing any problems before I even know we have one. :thumbsup

Thanks, Steve. I have always said that nobody can promise you perfection...what matters is how they handle it when the inevitable happens.

As always, we appreciate your trust, your business and your friendship.


--T

nothing beats 2 min server monitoring :)

one quick question is this a be up in 5 type of thing or a might take an hour thing i have backup plans for situations like this wondering if i should go to backup plan :)

I honestly do not know. The guys are working on it now and will let me know when it is corrected. I try my best not to bother them during an event like this unless it seems to drag on (which usually does not happen) so give it a few minutes and then if I haven't heard anything back then we will see what we can find out. Fair enough?

--T

ouch its down hard from Cali

i totally agree thats why i posted the question instead of bothering tech's with a ticket :)

will give it another 10 min's running late for dinner at a friends house.

We've been down for a complete hour :( Hope this is solved soon

you sure ? I count around 30 min :2 cents:

75 minutes

1 min server monitoring.... :2 cents:

it went down for us at 8pm on the dot.

Same here.
2 out of 3 boxes. Number 3 is still running
I am not worried though. I know they are doing everything within their power to fix things asap.

Now imagine if theplanet reps came offering "free" hosting while it gets sorted, and would move you back later.... Wonder how that would be looked upon :)

................

Down for hours vs down for days or weeks

Yeah, I can see the comparison.

UPDATE: Problem has been identified and isolated. Servers should start coming back up soon. I will have an update in a few minutes on how long it will take to all be back up.

Again, thanks for your patience and sorry about the issue.


--T

actually what matters is that the inevitable DOESN'T happen.

i told Matt that I'd be positive this week
but the affiliate program (mattsmoney.com)
has been down for 2 hours now and that network
has problems nearly every week.

Thankfully we only put the NATS
server over with natnet since
they do the NATS configuration free.

All our other servers are with more reliable
hosts like theplanet and cari.

If those guys go down
its because the server is dead
or there's a nuclear attack.

Anyways, good luck.

and PS the important thing
would be redundancy and uptime lol

Down for hours because of an "unknown" problem vs down for days because the fucking datacenter blew up with all your shit inside it?

Anyway, you took it as something "bad" no? The initial post at that time was intended to be good, why take the exact same thing as something bad now? I think you got the point. I got nothing gainst natnet, one of the better hosting companies, but that was not a classy move.

looks like we might be back up

UPDATE:


Tonight our central firewall device that updates all of the rules at the server level starting updating each server in the data center with bad code. The code had a bug in it that was undetected by our vendor or in our lab that basically told the machines to stop all traffic...period. Of course, it only took a few minutes to realize that there was a problem and stop the spread of the bad code. Once the bad code was isolated, we had to correct it and within minutes we were able to work out the issue with our vendor. Shortly after that we had the new code in the central device and it was propagating to all of the machines in the data center. The problem was that the machines with the bad code on them were told to stop all traffic, so they were not updating. We were forced to log into each machine individually and reset the rules and reboot the machines. Luckily there was only a hundred or so machines affected and we have a dozen System Adminstrators in the data center now and we are almost done.

Again, I apologize for this issue. Anyone that has been a customer for any length of time knows that these things rarely happen, but when they do you can count on us to be there and get it corrected quickly.

If you have any questions, comments or concerns please feel free to contact me or any member of my staff.

Warmest Regards,

Tony Morgan
CEO
NationalNet, Inc.

Tenth Floor Executive Offices
55 Marietta Street North West
Atlanta, Georgia 30303 USA

Telephone: 404-420-7000
Facsimile: 404-420-7010
Toll Free: 1-888-4-NATNET

Jordan's ass will be bouncing in celebration here any minute now!

http://www.stevelightspeed.com/jordan.gif

no wonder I couldn't access twistyscash..... I'll check again tomorrow .. I guess ..

good to know you guys are on it... been down over an hour over here now...

Well, even though we are now down about 230 joins for this time of day on a Thursday, I still love Natnet. Shit happens.

damn 230 joins thats some crazy bling.... im sure you'll find a way to make 230 more sales than u usually make fridays though

Awesome... Sweet T. We have been with NatNet for a while now (Years) and have only been down a couple times for a few minutes this was a first time this long.
Anyway no worries I wouldnt trust anyone else with my servers other then Nat Net.


I know your staff is the best:2 cents:

Wow those are nice numbers Steve, it's ashame you're down right now, best of luck.:helpme

I just want to reiterate that even though many of you already see servers coming back online to manually connect a monitor and keyboard to each server, log in, make the necessary changes and reboot the machine takes several minutes each. Even with a dozen people working on a couple of hundred servers it will take some time to get to every machine. We are working feverishly to get it fixed as fast as humanly possible and I can assure you we will make it right by our customers.

With that said, hopefully it will not be too much longer!

Again, I am deeply sorry for the outage and any inconvenience that it has caused you. We will have this behind us very soon, I promise.


--T

Natnet is all over it. They have some of the best tech support and network admins eva!!!!!!

Tada! :thumbsup

I have noticed quite a bit of timeouts, I thought it was my internet connection. Then I'd try to access JBM and a few other paysites, not realizing they were all hosted on Natnet, so I thought the problem was my ISP.
WG

Ryan does not speak for everyone at Matt's Money. I love NatNet and my experience in the past 3 years has been nothing been 100% satisfication. Bill V. is awesome, billing dept is great, and all the tech's are top notch...I have yet to meet SweetT though.

Chasin

Thanks for the kind words Chasin,

Unfortunately, because of the firewall issue (details are posted above) and the fact that it affected things at the server level, all the redundancy in the world would not have prevented this.

On the flip side, it's good to have so many people, both vendors and employees that we can call in during an emergency of this nature.

We're making good progress on this for sure.

Thanks

-- Bill

Matt should reign in the clown webmaster - taking a shot at NatNet is like taking a shot at Santa Claus - even if you're right it's not a smart thing to do.

How many more servers do you still have to go through?

To be honest, I'm not sure. I'm dealing with the tickets as they come in and letting my people do their job. As stated before, they have to physically hook up a keyboard and monitor to each affected server, flush the bad firewall rules and reboot it. It's a tedious process. There are about 15 techs in the data center right now and servers are coming up all over the place. I think they are less than an hour away from being done with touching all the servers and confirming that all is good.

Do you have servers with us? If so, if you would like to submit a ticket, I'll personally look into it.

EDIT - I just checked your profile to see if you had a domain listed in it (which you do) and it looks like your servers were not affected by this. If I am wrong, you can submit a ticket and again, I'll look into it.

-- Bill

my servers where effected but they are back up.. and I got some paysites on someone elses natnet server that came back up ~20-30 min after my last post

So now all up

FINAL UPDATE:

Sorry I am just now getting back to this thread but as of an hour or so ago 99.9% of the servers were back online and running normally. The final few are ones that are just being normally stubborn and should be up by this posting. If you are seeing anything abnormal please contact tech support asap.

Once again, I apologize to the customers that were affected by this situation. Please know that nobody wants you to have a better experience with NationalNet than I do. We will continue to do whatever it takes to insure that we are here for you through the good times and the bad.

As always, if you have any questions, comments or concerns please feel free to address them to me, directly, or to any member of my staff.

Warmest Regards,
Tony Morgan
CEO
NationalNet, Inc.

As a company that's had mission-critical servers with Planet (for about 4 years; we were with both EV-1 and Planet before they merged) and currently is with national net, I'd just like to reiterate that comparing Planet to NatNet is like comparing, oh, a Yugo with a Rolls Royce.

I won't go through all of the absolutely horrid experiences I had with Planet, but suffice it to say that before we were done, I had a cell phone number of their senior VP for technical services, and it STILL took hours to get stuff done. I believe that management genuinely wanted a reliable product, but was stuck with utterly inept staff... because apparently all the competent staff quit when the buyout happened.

Since we've been with NationalNet, the time I spend having to deal with admin issues is nearly zero, where it had been many hours per month with Planet. I truly believe that NatNet staff are not allowed to say "no" to a request; they are so generous and accommodating (and so quick to respond) that it is really beyond comparison.

Anyone who runs an online business gets pissed when servers go down because it is a direct hit to the bottom line, but I honestly believe that Bill, Tony and the rest of the NatNet crew do absolutely everything possible to solve problems faster than humanly possible when they do come up.

Can you imagine Planet being able to call its off-duty techs and vendor people and get them to come in on a Thursday night to fix customer problems? Not likely.