Tada! :thumbsup

I have noticed quite a bit of timeouts, I thought it was my internet connection. Then I'd try to access JBM and a few other paysites, not realizing they were all hosted on Natnet, so I thought the problem was my ISP.
WG

Ryan does not speak for everyone at Matt's Money. I love NatNet and my experience in the past 3 years has been nothing been 100% satisfication. Bill V. is awesome, billing dept is great, and all the tech's are top notch...I have yet to meet SweetT though.

Chasin

Thanks for the kind words Chasin,

Unfortunately, because of the firewall issue (details are posted above) and the fact that it affected things at the server level, all the redundancy in the world would not have prevented this.

On the flip side, it's good to have so many people, both vendors and employees that we can call in during an emergency of this nature.

We're making good progress on this for sure.

Thanks

-- Bill

Matt should reign in the clown webmaster - taking a shot at NatNet is like taking a shot at Santa Claus - even if you're right it's not a smart thing to do.

How many more servers do you still have to go through?

To be honest, I'm not sure. I'm dealing with the tickets as they come in and letting my people do their job. As stated before, they have to physically hook up a keyboard and monitor to each affected server, flush the bad firewall rules and reboot it. It's a tedious process. There are about 15 techs in the data center right now and servers are coming up all over the place. I think they are less than an hour away from being done with touching all the servers and confirming that all is good.

Do you have servers with us? If so, if you would like to submit a ticket, I'll personally look into it.

EDIT - I just checked your profile to see if you had a domain listed in it (which you do) and it looks like your servers were not affected by this. If I am wrong, you can submit a ticket and again, I'll look into it.

-- Bill

my servers where effected but they are back up.. and I got some paysites on someone elses natnet server that came back up ~20-30 min after my last post

So now all up

FINAL UPDATE:

Sorry I am just now getting back to this thread but as of an hour or so ago 99.9% of the servers were back online and running normally. The final few are ones that are just being normally stubborn and should be up by this posting. If you are seeing anything abnormal please contact tech support asap.

Once again, I apologize to the customers that were affected by this situation. Please know that nobody wants you to have a better experience with NationalNet than I do. We will continue to do whatever it takes to insure that we are here for you through the good times and the bad.

As always, if you have any questions, comments or concerns please feel free to address them to me, directly, or to any member of my staff.

Warmest Regards,
Tony Morgan
CEO
NationalNet, Inc.

As a company that's had mission-critical servers with Planet (for about 4 years; we were with both EV-1 and Planet before they merged) and currently is with national net, I'd just like to reiterate that comparing Planet to NatNet is like comparing, oh, a Yugo with a Rolls Royce.

I won't go through all of the absolutely horrid experiences I had with Planet, but suffice it to say that before we were done, I had a cell phone number of their senior VP for technical services, and it STILL took hours to get stuff done. I believe that management genuinely wanted a reliable product, but was stuck with utterly inept staff... because apparently all the competent staff quit when the buyout happened.

Since we've been with NationalNet, the time I spend having to deal with admin issues is nearly zero, where it had been many hours per month with Planet. I truly believe that NatNet staff are not allowed to say "no" to a request; they are so generous and accommodating (and so quick to respond) that it is really beyond comparison.

Anyone who runs an online business gets pissed when servers go down because it is a direct hit to the bottom line, but I honestly believe that Bill, Tony and the rest of the NatNet crew do absolutely everything possible to solve problems faster than humanly possible when they do come up.

Can you imagine Planet being able to call its off-duty techs and vendor people and get them to come in on a Thursday night to fix customer problems? Not likely.

Is the NatNet outage over now?

Thanks for your concern but the nightmare has been over for about 10 hours now. Sorry for the service interruption.


--T

ya downtimes like that one suck big time! thanks for update, i can start pushing traffic again! :winkwink: