Westhost sites down

[rise2it]

It appears anything hosted by Westhost has been down since about 3:30pm eastern time (roughly three hours now), and they are working on the problem.

[Demaestro]

The error message from my email program has recently changed from:

"Can't find address"

to:

"Connection is refused"

[Nativeozzie]

"I've got one WestHost hosted domain left that still isn't coming up, (web or email), the other 60+ sites have all come back."

I must be one of the "lucky ones", I bet they have completely lost my data.

[trillianjedi]

I bet they have completely lost my data.

That's impossible because of course you'll have off-site backups and local copies.

Herein lies a very important lesson. And it will happen again, too, just when you had started to forget about the pain of it all.

Been there. Done that. More than once.

I know how you feel - my thoughts are with you - hopefully will all come back shortly.

[simplesimon]

you should cut them some slack because something like a fire suppression system going off is a really big mess.

And to allow a circumstance where a vendor could accidentally discharge it in their data center is incompetance and a violation of their client's trust (rather than an excuse, as they have tried to use it). And to have the backups in the same room is negligence. Red flags all over. That's my opinon. I'm outta there.

[rise2it]

[members.westhost.com...]

Short version - they are now saying it could take as long as Friday before all servers are restored.

[willybfriendly]

As far as I can tell all my WH sites are up. But, I have received two emails this afternoon telling my that I have a site on an effected server, blah, blah...

For the life of me I can't figure out which of our sites might be on one of "12 shared and 6 dedicated servers" that are still not operational.

Good news is that they are sending out communications and are taking steps to let site owners get a temp site up and running.

The bad news is that it appears they are sending out emails to folks that don't need to hear it now, and, of course, that this will take a full week to get fixed.

Latest email states that they are "co-locating" at this data center. Anyone know the details of who "really" owns and runs the thing? It is possible that WestHost was effectively an innocent bystander when it all came down - depending on exactly what they are renting/contracting. (Of course that would suggest that their market copy is misleading at best!) I have been somewhat concerned since they began pushing vps.net as a replacement for their in-house vps solution.

[moose606]

I have had two ecommerce sites offline until 10 AM this morning. I have used westhost for 6+ years, but with this last issue, I am not sure if they are the same westhost as in past. Those of us looking for new hosting, what are are options? Who should we be looking at? What should be our new best practices when we migrate to new hosting? Might as well gain something from this experience.

[Leosghost]

They are owned by UK2GROUP..

[incrediBILL]

I pay extra for their nighly remote backup service.

Now you see how much that nightly backup service is actually worth during a mass outage.

Sure, they could fire up a new server and restore a backup, but there's probably thousands of dedicated servers waiting on the same restoration service and only so many people to partition and deploy the servers.

Hate to see people having these kinds of problems but it only takes getting massively burned once to come up with a better contingency plan and having all your eggs in a single host basket will never do anything but massively fail in the end.

[edited by: incrediBILL at 3:02 am (utc) on Feb 23, 2010]

[willybfriendly]

Interesting. Just had a look at UK2GROUP's website.

UK2, a "major player in the hosting industry since the late nineties", acquired Utah-based WestHost.com as the final part of "growth-by-acquisition" program designed to establish the company in the US hosting market. WestHost's data center in Salt Lake City and offices in Logan, Utah will continue to operate and UK2 Group will retain all "key personnel".

Explains the push for vps.net and a whole lot of doublespeak around that issue as well.

Is this common in the UK?

At least we know they aren't "too big to fail" :)

[Nativeozzie]

"That's impossible because of course you'll have off-site backups and local copies."

Let's see you download 50mb+ daily with download limits and see how fast you end up being put down to a dial up connection for the rest of the month. I pay them for a service and I am sick of the smug responses from people who blame me rather than the host.

[wheel]

Nobody's blaming you. They're telling you that backups of your data are ultimately, your responsibility. If you outsource it somewhere that doesn't look after it, or you don't steps to ensure backups are being handled properly, or if you don't spend the money to get the job done properly, and something screws up, nobody's going to blame you - but it's still your responsibility.

Learn as the rest of us have. Nobody's going to look after your backups better than you. Take an active hand in the process.

[lammert]

Latest email states that they are "co-locating" at this data center. Anyone know the details of who "really" owns and runs the thing?

The data center is owned by Consonus, and according to forum posts by official Westhost employees on the Westhost forums the test of the fire suppression system was conducted by Consonus, not Westhost. It seems Westhost was already planning to leave this data center sooner or later because it only contained their older servers and newer servers have been placed in other data centers in the past.

The fire suppression system used Inergen, a gas containing the three inert gases Nitrogen, Argon and Carbondioxide. Although everyone expected that these three inert gases have no influences on computer equipment and are therefore better than other fire suppression systems including Halon gases, it seems that there have been reports in the past that the release of Inergen can cause computer equipment to fail but real proof of that claim fails.

This kind of fire suppressing system is present in many--if not most--data centers. That it happened at Westhost seems just to be a case of bad luck. It could--and can--happen in other data centers of other providers.

[incrediBILL]

Let's see you download 50mb+ daily with download limits

Perhaps it's time to find a new vendor.

First, you can use software like RSYNC and only download the daily changes, not every file you own. Some tools can even download just the changes in files, not the entire file, so a growing database only needs the updates downloaded, not the entire database each time.

Second, most 50MB data when zipped is much smaller and faster to move maybe as small as 2-10MB in size.

Lastly, you can actually backup to other places, not your home using a restrictive ISP.

For as little as $99/mo or even less, you can have a dedicated server with terabytes of bandwidth where you can backup to daily.

Worse case, you can store 7GBs in gmail for free sent as a big (or multiple) file attachment(s) or unlimited storage on Yahoo Mail.

It's not a matter of whether it can be done, it's which method works best for a given situation and finding (or building) simple scripts and tools to make it happen.

Heck, most modern FTP clients even do file compression on the fly so 50MB isn't 50MB anymore.

In my case I download about 100MB from one site daily, zips down to 11MB, it rotates in 7 folders by day or week, then a baseline copy is set aside periodically.

Like I said before, not blaming people who got burned, having been a host I know how the best laid plans can go totally south, just offering options to help avoid pitfalls in the future.

That it happened at Westhost seems just to be a case of bad luck. It could--and can--happen in other data centers of other providers.

Yup.

I consider my current web host 2nd to none, they're top notch, but one of their data centers (where my big servers are) was hammered by a storm and failing generator once and we lost a few hours, then they were running electrical tests another time and triggered an electrical outage for a couple of hours, still not bad for a 5 year track record.

When I used to host in FL there was a major backbone router that blew in GA, host was up and running fine, site was down to everyone not in FL for 12 hours ;)

When I was a host, one fine morning L3 in So Cal had a single expensive part blow up and the only backup part was 3 hours away and they had to send someone to get it, 6 hours round trip, so most of So. CA (where our hosting facility was) was dead for a day. We were up and running fine, but the alternative routes were flooded with rerouting traffic so even backup data pipes were useless due to the nature and scope of the outage. All my customers were offline and it was nothing I did, nothing I had control over, but try to convince customers of that.

Just trying to give some insights into how anything can happen, and no matter how well you try to avoid it, a hosting failure will hit the fan eventually, it's inevitable.

Taking a week to restore service still seems a bit crazy.

[JS_Harris]

This is a good example of why it's a good idea to host a website with one company while managing domain DNS from a different company and pointing it at the current webhost.

When the host goes down you can move a copy of your site to a new host and still access DNS information so that the site resolves to that new host.

A week offline is too much for any webhost to expect to stay in business.

[willybfriendly]

FWIW, a site of mine at WH that made it through this unscathed just went down within the past 10 minutes or so. Perhaps this is the one they have been telling me is effected - even though it was chugging along without problems?

Can't even ping it for a response :(

Tracert shows it hanging at the WH datacenter.

[oddsod]

Now you see how much that nightly backup service is actually worth during a mass outage....having all your eggs in a single host basket will never do anything but massively fail in the end.

I don't have all my eggs in one hosting basket. I take great care to use multiple hosts. Even with the site in my signature it's only the forum that is with Westhost, the rest of the site is elsewhere.

One solution doesn't work for all webmasters. Some may have just one or two large sites in which case it's easier to create and manage emergency plans. Others like me earn from a portfolio of sites and deal with risk (and for IP reasons) by spreading the hosting around. Even among the large site brigade, the webmaster earning $10K a month from his site may choose a $99 a month solution that backs up evey night while the guy making just $500 may choose to do manual backups once a month. And some of them may not need any backups at all. One of my best sites is completely static and the copy on my hard disk (and my hard disk backups) is enough security for me.

Different strokes for different folks.

according to forum posts by official Westhost employees on the Westhost forums the test of the fire suppression system was conducted by Consonus

According to phone calls with Consonus, there's no way Inergen can cause these problems, it's been tested in DCs before. What they do accept is that if the environment is not up to standard - loose bits and pieces lying about the DC - the high pressure released by the fire suppresant can cause objects to move about and cause damage.

This kind of fire suppressing system is present in many--if not most--data centers. That it happened at Westhost seems just to be a case of bad luck. It could--and can--happen in other data centers of other providers.

This week of downtime is not bad luck, it's bad management. They go to great lengths on their site to explain how safe their DCs are - earthquake protection, power surges protection, power cuts plans, generators, several days worth of diesel in underground tanks etc. etc. But the realities emerging are that their protection and their recovery "plans" have as many holes as a roll of swiss cheese.

Everybody takes ultimate responsibility for their own data, yes, but when a host has screwed up, the host has screwed up. Screwed up not just on planning for the contingency but on the subsequent recovery. Storing backup servers in the same DC is but one example of their ineptitude. Westhost owners editing the Wikipedia entry in the last few hours to try and cover up their mismanagement doesn't do them any credit either.

[wheel]

but when a host has screwed up, the host has screwed up.

And hosts will screw up. You have my personal guarantee on that.

That's why a proper contingency plan doesn't include 'backing up to my existing hosting company'. They can burn my hosting company to the ground with my server and data in it and I'm down for about a day, maybe less. I have hardware and backups located in other locations. I am not completely reliant on a single point of failure as best I'm able.

[oddsod]

I refuse to accept that every single host out there is capable of lying, stringing customers along pretending there is no problem/it's a minor problem and increasing overall distress by ignoring genuine concerns and refusing to say what's going on or even what ETA they're working towards.

Westhost's sister company (also owned by UK2Group) is an example of how it should be done. They too were affected at this DC but communicated to their customers within two hours that 500 hard disks had gone down and what they were doing about it. They also maintained a status page with real information, not the phone-hold "your business is important to us, please wait" nonsense. After several days of their sites being down, it's not surprising that a lot of WH customers were getting tired of being on hold.

[incrediBILL]

But the realities emerging are that their protection and their recovery "plans" have as many holes as a roll of swiss cheese.

It looks like the reality is they never planned for what happens after the fire suppression system goes off - something they're learning now the hard way.

Let's just say, with all puns intended, they're getting a real life fire drill :)

[rise2it]

"And hosts will screw up. You have my personal guarantee on that. "

Agree 100%

-------------

"I refuse to accept that every single host out there is capable of lying, stringing customers along "

I've come to the conclusion web hosts are like merchant credit card processors - all pretty much the same, and to not expect service from any of them.

[bwnbwn]

rise2it

I've come to the conclusion web hosts are like merchant credit card processors - all pretty much the same, and to not expect service from any of them.

Old saying everybody has their opinion. I beg the differ. I have excellent host and great customer service that I expect. I excpect great customer service from my employees to my customers. Placing all them in the same basket is not my opinion and I bet many here will state the same.

And hosts will screw up. You have my personal guarantee on that.

,and you have my personal guarantee on this YOU and I will as well.

I suppose when machines take over for humans then we will live in a perfect world, until then expect the unexpected and have a plan established.

[iJeep]

Luckily I moved my "bread & butter" site from WH years ago (to Pair, which has been flawless).

I still have a reseller account with WH for smaller sites, which are still important, but not my livelihood.

They have only delivered poor support and plenty of downtime. I'm moving that reseller account now, as different sites come back on-line from Westhost.

While tarring the files, I have noticed that the servers are horribly slow, even after coming back on-line. I spent 4+ hours last night waiting on a site to archive, which took less than one minute to decompress on the new host.

There are still major issues at WH, I feel this is far from over.

[Nativeozzie]

I'll let you know how mine goes, if it ever comes back up. Apparently my server is being left until last it seems.

[oddsod]

and you have my personal guarantee on this YOU and I will as well.

I won't. Bear in mind I was originally talking about what Westhost did and how they responded, not what happened to their DC.

[Hollywood]

Of course my PRIMARY money making site is down for days, I have no new deals since last Friday, this is really hurting, still down 100% and I hope like hell this wont take till Friday to fix cause this REALLY SUCKS!

Not sure what to do as we do have database on Westhost end and I am afraid of moving everything to new host and still not having our realtime client data base, not to mention how the search engines will handle this; if I move I am afraid my serps will goto hell.

Hollywood

[David Bruning]

I've switched to their 4.0 servers and dns is perculating now. Also signed up for a rackspace account which is active now. I'll give Westhost a chance to make this right, but if they don't do a better job of communicating, next time my personal sites as well as the work ones are hopping.

[rise2it]

bwnbwn, I know I sounded a bit negative with my hosting comment.

However, I think you'll also agree that you actually find out how good any company is WHEN they screw up.

Service looks great from anyone as long as things are going smoothly.

As for Westhost, I'm sure they are doing their best, but they also handled things poorly (as far as annoucements). I waited 12 hours to switch, and thank goodness I did. Unfortunately, some here kept believing their ETAs and did not.

Also, I did not receive the first email from them (to an address that has nothing to do with Westhost) until last night. I (and everyone else affected) SHOULD have been notified during the first few hours.

Had I not been using Saturday afternoon to do some 'catching up' (and I just happened to be online when everyone went down), I would not have known there was a problem until Sunday night or Monday morning when I logged in to look at weekend sales.

Once they realized the severity of the problem, they should have been emailing their customers. Actually, they should have had staffers CALLING their customers, especially those paying for larger accounts and/or dedicated servers.

In all honesty, I'll probably point back to Westhost when all is said and done, as their uptime for the three of so years we've had them has been virtually perfect, and the few times I've had a question, it was handled quickly (the reason I left my last host for my main site).

I understand they were overwhelmed, and hopefully they'll learn from it.

[bwnbwn]

rise2it I hope we all learn from this.

I can understand your feelings and am glad to see your cooling off and seee the positive parts of this company. This was a tragic mistake one I am sure no host has prepared themselves for.

[incrediBILL]

Actually, they should have had staffers CALLING their customers, especially those paying for larger accounts and/or dedicated

The flip side to that argument is the staffers should be working to restore service faster instead of wasting their time on the phone.

As long as someone posts a message on their web site and changes the voice mail message to let me know they are aware the servers are down and working on them, I'd prefer all hands working to restore service instead of answering the phone.

In reality, customers with down servers, especially when it's known to be a larger issue, are often their own worse enemy during a crisis.

When you're demanding to waste the time of the staffers by whining on the phone instead of letting the staffers work to restore their service, you've become part of the problem, and people just don't get it.