Westhost sites down

[rise2it]

It appears anything hosted by Westhost has been down since about 3:30pm eastern time (roughly three hours now), and they are working on the problem.

[bobothecat]

My server, for example, has been ping-able throughout the issue, but you can't actually get to it.

It's really strange. My one site that is affected mostly shows either 404's or 403's, but I'm still able to retrieve some pages with no problem... oddest thing I've ever seen.

[Nativeozzie]

I actually registered on Webmasterworld to find a new host, I was only with Westhost because years ago webmasterworld had a link to them as their host.. now I find you don't seem to be able to find that sort of info here.

[willybfriendly]

It seems they are making progress. Had two sites that were apparently unaffected. Another (e-commerce) was back on line this morning, but I had to repair a couple of DB tables (most likely due to visitors being onsite when everything crashed). A third has come back on line in the past couple of hours.

I can be pretty forgiving of hardware failures. S**t happens, and I don't expect them to have a semi truck load of redundant equipment on site. Shucks, I have seen a major State govt agency lose their system for two days over a single part that had to be shipped in.

Lack of transparence when things go wrong is troublesome.

[Nativeozzie]

They have now added the following to their outage info..


"The network service status listed below is inaccurate due to hardware failures."

[rise2it]

They have now added the following to their outage info..


"The network service status listed below is inaccurate due to hardware failures."

---------------------

Maybe one of them is reading this thread :-)

Again, I think they should have posted that YESTERDAY.

The problem isn't that they had a situation....the problem is in how poorly they have handled the situation.

It's good to hear thoughthat someone's site has come back to life.

[bobothecat]

"The network service status listed below is inaccurate due to hardware failures."

They should have the same disclaimer for the "Fix ETA".

If you want my personal opinion... I think they were hacked.

Yesterday evening when the outage occured, I noticed that my SecureCRT connection disconnected... after trying to re-connect, it said the password was invalid, asking for me to type it in again (which I didn't), that's when I knew something was wrong and called.

Thus far the server is still reachable by ping and traceroute... kind of odd since it's supposed to be 'offline'.

Some page requests still come up, while others don't (as mentioned earlier).

I'm posting this now, only because after spending nearly 24 hours talking to 'technical support' asking to speak to an actual tech. I've been refused.

[edited by: bobothecat at 12:20 am (utc) on Feb 22, 2010]

[rise2it]

Going through the Westhost forum (on their site), it seems everyone that calls in is being told (and has been told) something different.

[simplesimon]

They added 12 hrs to the eta again - back to 8AM (tomorrow now). Already moved one website today. Guess I'll be moving the rest. I have been stalling on moving my email off of their servers. I just hope their backups are intact so I can get the 2 days worth of email.

[Lorel]

My site has been down for over 24 hours except for a brief respite when I got in some email this morning. I have 35 clients on Westhost, mostly on different servers and some of those sites are up- although their email/carts/etc may not be working.

I'm not about to switch from Westhost however as it's the best host I've every used. They are constantly updating their Software/hardware so I'll just trust they get this worked out quickly.

[Nativeozzie]

I think it's long past getting worked out quickly don't you?

[rise2it]

Well, if the reason was the tester from the company that installed/maintained the equipment that caused the accident and set the ball rolling, then imagine the damage/stress caused to the Westhost people.

Again, though - with things this far out of control, they should have notified everyone THEN so we could have started making other arrangements, because they had to know THEN things were not going to be fixed in three hours.

I did trust my gut and switched early this morning, but the 12 hours I waited because they chose not to be honest about how bad things were cost me (just as it has cost a lot of you) a lot of business.

Now, I'm not the most tech savvy guy on here, but now that we're looking at 36+ hours by in the morning, is it not possible for them to somehow stick up a page on the affected domains so visitors aren't getting 403/404 errors and thinking we are no longer in business?

What are returning/repeat customers going to think? What is the customer that had a problem with an order and tried to contact us over the weekend going to think?

Their customer's first assumption is going to be that WE have gone under.

Again, this is where Westhost needed to STEP UP.

[lammert]

My experience is that technicians are optimists by nature and they just hadn't any idea of the size of the problems at the beginning. This looks like a Rackspace problem a few months ago where a transition from normal to UPS power caused havoc. Instead of just restarting the servers, they had to replace a few hundred power supplies, routers, switches and other networking equipment before everything was running again.

An estimation of the size of the damage and the needed action and the time frame for repair is often subject to iteration in this kind of situations. But I agree that they should be able to make a good estimation of the ETA now and communicate it to their customers.

[Nativeozzie]

I have zero doubt there will be a further ETA added when it starts getting close to 8am, currently have lost 2 full days trading and are unable to ship out previous orders or communicate with customers at all.

[CernyM]

Ours finally came back up and I've heard others have come back up through the day. Looks like its just a long slog.

[Nativeozzie]

Ugh, why the heck can't mine come back up.

[oddsod]

This a good example of how not to handle things: lie to your customers, keep changing the ETA, don't answer any questions, don't provide any information...and when you do post information (example thread [forums.westhost.com]) make sure it's virutally useless corporate speak: "We value your custom and are working hard blah blah."

All my accounts with them are still down, including multiple dedicated servers. Today is the third day. And no idea when they'll be working again.

I have zero doubt there will be a further ETA added when it starts getting close to 8am, currently have lost 2 full days trading and are unable to ship out previous orders or communicate with customers at all.

Well that's if they can even be bothered to update the status page any more. It's a few minutes away from that ETA right now and there's not the smallest glimmer that my sites are back online.

[cmendla]

I guess it could be worse. The problem I have is that the email I use for almost all of my accounts in on a site that is currently down due to the WH thing.

I'm hoping that it will come back up some time this morning.

[bobothecat]

It's a few minutes away from that ETA right now and there's not the smallest glimmer that my sites are back online.

I'm sure they'll just change it to 8:00 PM... and continue with the same old babble as usual.

[simplesimon]

If there are just a handful of sites left I doubt they are going to sweat it. About a month ago I had an IP stop responding during launch on Friday night and the "techs" on weekend duty had no clue. Nothing was done about it until the admins came in on monday morning, and when they finally fixed it they didn't bother to tell me. Of course I had launched elsewhere by that time. Westhost sure gives you good practice in moving and restoring websites ;)

[oddsod]

Tell me about it!

Apparently, and according to tech support, the backup servers are in the same datacenter!

That's not what they told me when they sold me the service of nightly backups for dedicated servers. They sold it on the merits that as it was in a different location it was a lot safer.

ETA has passed and the latest news is that they don't have a new ETA.

[Demaestro]

All my sites are up from them.

One of my email servers is still down. At least my sites are up, I didn't see any interruption to any of my sites, but I still don't have my top level email.

Totally unrelated to this issue I have been planning a move from them in the next couple months.

I may speed up my plans though and get everything I have moved out of there sooner than I wanted to. My reason was a recent price change in the wrong direction and a very good local quote, but reading this thread and a weekend without email has made me rethink my time line.

A few years ago they were one of the only places willing to let me have mod_python on a non-dedicated server and for that I loved them, unfortunately the shine has worn off.

[edited by: Demaestro at 4:21 pm (utc) on Feb 22, 2010]

[oddsod]

Even better, they claim that 98% of their customers are back online (up from 95% earlier).

They've now got 75%(?) of customers all bemoaning the fact that they're in (what they think is) the non-working 2%.

[justgowithit]

My sites were all back online yesterday. I've got sites hosted on several different servers with them. Some were unaffected, others weren't so lucky.

[blend27]

-- weren't so lucky --

Why?

[simplesimon]

I have seven or eight clients with their own westhost accounts, and my own reseller account, and none of those were affected. The one account that is still down has certainly kept me busy through the weekend though.

[incrediBILL]

Having been a host in a past life you should cut them some slack because something like a fire suppression system going off is a really big mess.

People complaining they're "out of business" today only have themselves to blame for not having a backup server or some other contingency plan.

What if Westhost were hit by a tornado or a flood or some other natural disaster?

You need to be prepared for the inevitable, not Westhost, as it's your business.

I have servers in 2 different data centers and a clone at home, so unless some supreme deity with a sense of humor knocks out both data centers in extremely opposite geographical locations, I'm prepared, are you?

[oddsod]

I'm sure they'll just change it to 8:00 PM... and continue with the same old babble as usual.

They did change it to 8 PM.

Now they've changed ETA to "we don't have a clue". (more precisely: Fix ETA: TBD)

I have servers in 2 different data centers and a clone at home

Well done. We can't all do that or want to do that. I pay Westhost for a dedicated server. I pay extra for their nighly remote backup service. I decided that that was a sufficient level of backup for what this site was worth to me. Problems happen, but it's how you deal with it that upsets people.

They screwed Brett Tabke around about five years ago (I take it he didn't have a spare copy of his site under his bed).

[rise2it]

The choice is up to you, but I chose on late Sat/early Sunday to not trust this would be a quick fix, and pointed the DNS records to a backup server.

While we've still lost a LOT of business (because of the time it takes the Domain Name records to update), we're not dead in the water. (Luckily, we don't have our online databases there, as some of you did...so it's basically just webpages and email).

For those sitting on the fence, I'd recommend grabbing some cheap hosting elsewhere, and uploading your files (at at least one static page for your customers to see even if you for some reason do not have your own backup files). Once you change the DNS entries, you'll start to get some business trickling in within 3 or 4 hours.

Once Westhost gets this fixed, you can then point your stuff back to their servers.

My BIGGEST gripe with them isn't that the site is down, but that the results they are serving are still 403/404....surely they could have at least routed something for our affected domains so at least the customer would see a webpage letting them know there was a tech issue, instead of the customer seeing nothing and thinking we were no longer in business - and I have spent today fielding THOSE types of calls.

There have been a couple of comments on other forums that some of the sites have DID get restored are now down AGAIN, and unfortunately I think this situation will continue to play out this way over the next few days, and maybe longer, as they swap hardware, restore backups, etc.

[Demaestro]

bill you raise excellent points.

What is really crappy though is that most of the people who have an account with Westhost have a backup plan with westhost.

I have backup files of all my sites locally so I can get any site back up all I need to do is wait for DNS to resolve.

What about for my IMAP email though? They claim to do an offsite backup? Why isn't that offsite backup online?

Is there anything I can do in the future to give me more control over my IMAP email so that if something like this happens that I am not missing 2 days worth of emails?

[lexipixel]

I've got one WestHost hosted domain left that still isn't coming up, (web or email), the other 60+ sites have all come back.