Major Hosting Problems - Help Please - GoFuckYourself.com

Socks · 05-05-2008, 08:50 AM

Ok, so I have 4 boxes at a popular host here, I won't name them now because I'm not trying to start a witch hunt or anything but here's my concern.

Since starting with this host, I can count at least 3 times that our boxes have been rebooted due to power loss, in the span of about 4 months.

I only ever get a shrug and a sorry, but it's seriously causing problems.

This last time our storage box came back up, but its raid mount was unmounted, and needed to be FSCK'd (again) which takes a few hours.

We also lost our NFS mounts on all the other machines that access this storage box, and they had to be removed and re-added manually. Our boxes are managed at this host just to be on the safe side, and their tech was experienced in BSD, fixed the raid, fixed the NFS mounts, and even helped us speed up one of the dual processor machines. I've been very happy with their service. I'm semi-needy, and they always help me out, and have yet to charge me extra for anything I've asked for, even things I would expect them to charge for.

However.. Boxes rebooting in the middle of the night, not cool. We don't have startup scripts setup on all the boxes quite yet, it's a real pain in the ass.

So to be clear, does this happen at all hosts? Should my boxes EVER lose power? Seems crazy to me.

PS: This last time we had no clue the boxes even went down, except that shit wasn't working right.. They came back up, but things weren't right, as I mentioned above.. No phone call, no e-mail, we had to find out ourselves and wonder WTF is going on.. When I saw /raid/ wasn't there I almost swallowed my head.

Va2k · 05-05-2008, 08:56 AM

No it shouldnt happen.. I have 5 boxes all over different hosting companies, Let me ask this though does the hosting company ya talking about start with "rig" ? if so lets chat

they are the newest for me and had a few ups and downs so far.

Quote:

Originally Posted by Socks

Ok, so I have 4 boxes at a popular host here, I won't name them now because I'm not trying to start a witch hunt or anything but here's my concern.

Since starting with this host, I can count at least 3 times that our boxes have been rebooted due to power loss, in the span of about 4 months.

I only ever get a shrug and a sorry, but it's seriously causing problems.

This last time our storage box came back up, but its raid mount was unmounted, and needed to be FSCK'd (again) which takes a few hours.

We also lost our NFS mounts on all the other machines that access this storage box, and they had to be removed and re-added manually. Our boxes are managed at this host just to be on the safe side, and their tech was experienced in BSD, fixed the raid, fixed the NFS mounts, and even helped us speed up one of the dual processor machines. I've been very happy with their service. I'm semi-needy, and they always help me out, and have yet to charge me extra for anything I've asked for, even things I would expect them to charge for.

However.. Boxes rebooting in the middle of the night, not cool. We don't have startup scripts setup on all the boxes quite yet, it's a real pain in the ass.

So to be clear, does this happen at all hosts? Should my boxes EVER lose power? Seems crazy to me.

PS: This last time we had no clue the boxes even went down, except that shit wasn't working right.. They came back up, but things weren't right, as I mentioned above.. No phone call, no e-mail, we had to find out ourselves and wonder WTF is going on.. When I saw /raid/ wasn't there I almost swallowed my head.

rowan · 05-05-2008, 09:15 AM

Are you sure it's power loss, and not an OS panic?

Kroy · 05-05-2008, 09:22 AM

That should not happen - it appears the hardware in your box(es) might be faulty.
New servers might really be in order...

Kroy · 05-05-2008, 09:23 AM

BTW, if it's really a true power loss and ALL your boxes were affected then that's a problem on the host's side. In that case switching to a different host might be in order.

Socks · 05-05-2008, 10:13 AM

Hey everyone, all power loss.. Two times electrical problems, another time was a "scheduled" reboot for some kind of port audit or something.

It's the electrical problems I'm concerned about. Not sure why our boxes should be turned off for ANY reason. Even if there are electrical issues that aren't immediate, shouldn't a good host be able to re-route our power before they mess with something?

Has this happened at other big adult hosts? We've been hosting websites for years and I've never heard of electrical problems that I can think of ever.

rowan · 05-05-2008, 10:21 AM

Have they offered a credit? Do they have an SLA?

I had electrical problems with one host and they ended up crediting me back half my monthly fee, even though their SLA technically said they only needed to credit me something like 15%. In that instance it was due to a faulty power controller. No such problems since, or prior for that matter. That's the only time in ~7 years of having dedicated boxes sprinkled around the globe that I have experienced issues with power.

rowan · 05-05-2008, 10:25 AM

Re-reading your first msg, you need to find out what monitoring and escalation procedures they have in place. A good host should be the one telling you that your servers are down, not the other way around.

You should also look at getting your startup in order so that a reboot will not require human attention (assuming the fsck doesn't require human intervention)

Ycaza · 05-05-2008, 10:26 AM

Its power, breaker going out, probably. I would doubt its the entire data center, the charging of fees reallly depends on the agreement you made. Is it fully managed, sounds like thats what you need. As a rule I do managed on an multiserver environment. Start up scripts, that should have beent aken care before, if not at least after the first power problem. Clients don't always let you know what they know, so the first one is excusable, but on the the first one then the host knows what they need to do.
someone will invariably come in here and shout a lot of BS about how its all supposed to be this or that, and it will all be BS. What I am telling you is realistic. 12years in the isp/hosting business realistic. You can pay lots to some and get fully managed to the keystroke and that can be a cool way to go, you can get fully managed where you have all the help you need, you can get dedicated where generally they charge you by incident.
I am 99 percent sure I know the company you are talking about, and I won't out them either.
Good luck either way, If you want some real help, my icq number is below.

Evil E · 05-05-2008, 10:43 AM

You shouldn't have any power failure at any good data center. One of the main obligations of hosting companies is to have very redundant electricity. Never had a problem myself but I don't do any hosting at a "adult host". If you do a lot of business with your hosting co, a cool thing to do is to take a tour of their data centers and operations.

"However.. Boxes rebooting in the middle of the night, not cool. We don't have startup scripts setup on all the boxes quite yet, it's a real pain in the ass."

-Like other said, before putting a box in production, you should make sure that if it reboots everything is fine and all the scripts load starting whatever services you need if it happens to reboot.

"PS: This last time we had no clue the boxes even went down, except that shit wasn't working right.. They came back up, but things weren't right, as I mentioned above.. No phone call, no e-mail, we had to find out ourselves and wonder WTF is going on.. When I saw /raid/ wasn't there I almost swallowed my head."

You should have notification traps setup, so that when any service or box is unreacheable you get an email, or you can even do some more funky custom scripts stuff like getting a call with nagios for example.

Best of luck! Get your startup scripts and notification traps set up ASAP

mike-al · 05-05-2008, 11:15 AM

I think i know who you be talking about, I have had this problem and not even been there a month yet and somedays I wake up and ALL machines rebooted...

I am going to inquire

This is not good PR press :p

Wiredoctor · 05-05-2008, 12:12 PM

Absolutely not shopuld you ever have a power loss, and especially a few of them??? Not a solid host if you ask me.

rowan · 05-05-2008, 04:34 PM

Assuming the link in Socks' sig is his site, it's not that hard to figure out the host.

Socks · 05-05-2008, 04:43 PM

fail rowan, not the same host

dubsix · 05-05-2008, 09:26 PM

no, you should never have those sort of power problems....

spooky181 · 05-06-2008, 07:51 PM

Who is the host I wanna know....

marketsmart · 05-06-2008, 08:03 PM

what caza said... i bet its a breaker... ask your host how many amps in your cab and how many servers are in the cab and ask them if they have done a power audit in your cab to make sure you arent riding aroun d the threshold...

_Rush_ · 05-06-2008, 08:46 PM

Names!

10 char

Theo · 05-06-2008, 09:02 PM

3 times in 4 months is unacceptable. I can imagine happening once every couple of years and under very certain circumstances.

Rochard · 05-06-2008, 11:33 PM

This is why I host with National Net. They are top notch all the way around and have the best support in the industry.

sicone · 05-07-2008, 12:01 AM

Quote:

Originally Posted by Rochard

This is why I host with National Net. They are top notch all the way around and have the best support in the industry.

I have been happy as can be since switching over to Nat Net

05-05-2008, 08:50 AM	#1
Socks Confirmed User Industry Role: Join Date: May 2002 Location: Toronto Posts: 8,475	Major Hosting Problems - Help Please Ok, so I have 4 boxes at a popular host here, I won't name them now because I'm not trying to start a witch hunt or anything but here's my concern. Since starting with this host, I can count at least 3 times that our boxes have been rebooted due to power loss, in the span of about 4 months. I only ever get a shrug and a sorry, but it's seriously causing problems. This last time our storage box came back up, but its raid mount was unmounted, and needed to be FSCK'd (again) which takes a few hours. We also lost our NFS mounts on all the other machines that access this storage box, and they had to be removed and re-added manually. Our boxes are managed at this host just to be on the safe side, and their tech was experienced in BSD, fixed the raid, fixed the NFS mounts, and even helped us speed up one of the dual processor machines. I've been very happy with their service. I'm semi-needy, and they always help me out, and have yet to charge me extra for anything I've asked for, even things I would expect them to charge for. However.. Boxes rebooting in the middle of the night, not cool. We don't have startup scripts setup on all the boxes quite yet, it's a real pain in the ass. So to be clear, does this happen at all hosts? Should my boxes EVER lose power? Seems crazy to me. PS: This last time we had no clue the boxes even went down, except that shit wasn't working right.. They came back up, but things weren't right, as I mentioned above.. No phone call, no e-mail, we had to find out ourselves and wonder WTF is going on.. When I saw /raid/ wasn't there I almost swallowed my head. Last edited by Socks; 05-05-2008 at 08:53 AM..

05-05-2008, 09:22 AM	#4
Kroy Confirmed User Industry Role: Join Date: Sep 2005 Location: Phoenix, AZ Posts: 413	That should not happen - it appears the hardware in your box(es) might be faulty. New servers might really be in order... __________________ Kroy The Invisible Hand Say Hello "The stone age didn't end because they ran out of stones..." - S. Yamani

05-05-2008, 09:23 AM	#5
Kroy Confirmed User Industry Role: Join Date: Sep 2005 Location: Phoenix, AZ Posts: 413	BTW, if it's really a true power loss and ALL your boxes were affected then that's a problem on the host's side. In that case switching to a different host might be in order. __________________ Kroy The Invisible Hand Say Hello "The stone age didn't end because they ran out of stones..." - S. Yamani

05-05-2008, 10:26 AM	#9
Ycaza Confirmed User Industry Role: Join Date: Feb 2005 Location: the 805 Posts: 4,290	Its power, breaker going out, probably. I would doubt its the entire data center, the charging of fees reallly depends on the agreement you made. Is it fully managed, sounds like thats what you need. As a rule I do managed on an multiserver environment. Start up scripts, that should have beent aken care before, if not at least after the first power problem. Clients don't always let you know what they know, so the first one is excusable, but on the the first one then the host knows what they need to do. someone will invariably come in here and shout a lot of BS about how its all supposed to be this or that, and it will all be BS. What I am telling you is realistic. 12years in the isp/hosting business realistic. You can pay lots to some and get fully managed to the keystroke and that can be a cool way to go, you can get fully managed where you have all the help you need, you can get dedicated where generally they charge you by incident. I am 99 percent sure I know the company you are talking about, and I won't out them either. Good luck either way, If you want some real help, my icq number is below. __________________ Caz Thrush Head Honcho [email protected] http://thrushtech.com ICQ: 304883574 do people still icq?

05-05-2008, 10:43 AM	#10
Evil E Confirmed User Join Date: Apr 2005 Location: Lazyness is a lifestyle Posts: 3,201	You shouldn't have any power failure at any good data center. One of the main obligations of hosting companies is to have very redundant electricity. Never had a problem myself but I don't do any hosting at a "adult host". If you do a lot of business with your hosting co, a cool thing to do is to take a tour of their data centers and operations. "However.. Boxes rebooting in the middle of the night, not cool. We don't have startup scripts setup on all the boxes quite yet, it's a real pain in the ass." -Like other said, before putting a box in production, you should make sure that if it reboots everything is fine and all the scripts load starting whatever services you need if it happens to reboot. "PS: This last time we had no clue the boxes even went down, except that shit wasn't working right.. They came back up, but things weren't right, as I mentioned above.. No phone call, no e-mail, we had to find out ourselves and wonder WTF is going on.. When I saw /raid/ wasn't there I almost swallowed my head." You should have notification traps setup, so that when any service or box is unreacheable you get an email, or you can even do some more funky custom scripts stuff like getting a call with nagios for example. Best of luck! Get your startup scripts and notification traps set up ASAP __________________ A girl once told me "Give me 8 inches and make it HURT". So, I fucked her twice and hit her with a brick.

05-05-2008, 09:15 AM	#3
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Are you sure it's power loss, and not an OS panic?

05-05-2008, 10:13 AM	#6
Socks Confirmed User Industry Role: Join Date: May 2002 Location: Toronto Posts: 8,475	Hey everyone, all power loss.. Two times electrical problems, another time was a "scheduled" reboot for some kind of port audit or something. It's the electrical problems I'm concerned about. Not sure why our boxes should be turned off for ANY reason. Even if there are electrical issues that aren't immediate, shouldn't a good host be able to re-route our power before they mess with something? Has this happened at other big adult hosts? We've been hosting websites for years and I've never heard of electrical problems that I can think of ever.

05-05-2008, 10:21 AM	#7
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Have they offered a credit? Do they have an SLA? I had electrical problems with one host and they ended up crediting me back half my monthly fee, even though their SLA technically said they only needed to credit me something like 15%. In that instance it was due to a faulty power controller. No such problems since, or prior for that matter. That's the only time in ~7 years of having dedicated boxes sprinkled around the globe that I have experienced issues with power.

05-05-2008, 10:25 AM	#8
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Re-reading your first msg, you need to find out what monitoring and escalation procedures they have in place. A good host should be the one telling you that your servers are down, not the other way around. You should also look at getting your startup in order so that a reboot will not require human attention (assuming the fsck doesn't require human intervention)

05-05-2008, 11:15 AM	#11
mike-al Confirmed User Join Date: Aug 2007 Posts: 691	I think i know who you be talking about, I have had this problem and not even been there a month yet and somedays I wake up and ALL machines rebooted... I am going to inquire This is not good PR press :p __________________ Delete this account, i am done here

05-05-2008, 12:12 PM	#12
Wiredoctor Confirmed User Join Date: Dec 2001 Location: The Great USA Posts: 1,632	Absolutely not shopuld you ever have a power loss, and especially a few of them??? Not a solid host if you ask me.

05-05-2008, 04:34 PM	#13
rowan Too lazy to set a custom title Join Date: Mar 2002 Location: Australia Posts: 17,393	Assuming the link in Socks' sig is his site, it's not that hard to figure out the host.

05-05-2008, 04:43 PM	#14
Socks Confirmed User Industry Role: Join Date: May 2002 Location: Toronto Posts: 8,475	fail rowan, not the same host

05-05-2008, 09:26 PM	#15
dubsix Confirmed User Industry Role: Join Date: Dec 2004 Posts: 363	no, you should never have those sort of power problems....

05-06-2008, 07:51 PM	#16
spooky181 Confirmed User Join Date: Jul 2003 Location: back of beyond Posts: 2,951	Who is the host I wanna know....

05-06-2008, 08:03 PM	#17
marketsmart HOMICIDAL TROLL KILLER Industry Role: Join Date: Dec 2004 Location: Sunnybrook Institution for the Criminally Insane Posts: 20,419	what caza said... i bet its a breaker... ask your host how many amps in your cab and how many servers are in the cab and ask them if they have done a power audit in your cab to make sure you arent riding aroun d the threshold...

05-06-2008, 08:46 PM	#18
_Rush_ Confirmed User Join Date: Dec 2006 Location: Buenos Aires Posts: 742	Names! 10 char __________________ No sig.

05-06-2008, 09:02 PM	#19
Theo HAL 9000 Industry Role: Join Date: May 2001 Posts: 34,515	3 times in 4 months is unacceptable. I can imagine happening once every couple of years and under very certain circumstances.

05-06-2008, 11:33 PM	#20
Rochard Jägermeister Test Pilot Industry Role: Join Date: Dec 2001 Location: NORCAL Posts: 73,998	This is why I host with National Net. They are top notch all the way around and have the best support in the industry. __________________ “The choice is no longer between right or left. The choice is between normal and crazy.” - Sarah Huckabee Sanders YNOT MAIL \| THE BEST ADULT MAILING SOLUTION