Westhost sites down

[rise2it]

It appears anything hosted by Westhost has been down since about 3:30pm eastern time (roughly three hours now), and they are working on the problem.

[CernyM]

Still down. They've been giving progressively worse ETA on fixes all day long.

I can ping my server, it just won't load.

This is the second time in the last three days that they've had an outage.

[herb]

10:30 PM pacific still down...

[fatpeter]

Still down 11am GMT

[rise2it]

Well....I'm a pretty understanding guy, and IF they had told the truth yesterday (instead of the 3 hour crap), we could have made better decisions on how to handle it.

It doesn't bother me it's down - it does bother me they didn't notify us or tell us what really is going on.

Now, I'm not the most tech savvy guy, but our site hasn't been down more than about 2 hours TOTAL (with other hosts) in the last decade combined, and that included my host getting hit with back to back hurricanes and having to fall back to backup generators.

In 1996, I can see this happening...in 2010, being down 16 hours straight is pretty much unacceptable.

[rise2it]

At midnight they changed their posting to 'hoping to have the issued fixed at 8am Mountain Time', so it appears they simply gave up and went home for the night.

When that time rolls around, I expect to see a message bumping it to further into the day.

[jdMorgan]

One site I track has already been restored for at least an hour. I can't tell exactly, because most visitors to this site aren't awake yet. But msnbot hit the site at 14:02 CDT (7:02AM at WestHost).

Jim

[herb]

8:05 Mountain Time
Just spoke with support, they are going to be moving the published ETA of 8am Mountain Time fwd.

8:00 p.m. MDT is now the posted ETA at:

[netstatus.westhost.com...]

[rise2it]

That's good to hear, Jim....gives me a little hope, anyway.

According to their status page, it's only about 5% of their servers that have had this issue, as the rest appear to have been up this entire time.

[CernyM]

It's more like 90% of their customers, according to the folks over there I've talked to.

Pretty much a disaster, and not the first time they've done this.

[herb]

The "status page" is interesting only for the ever moving ETA. We can't access our account and I was told "no one can". Seems all of the databases and management functions are on servers that are down. Email is about the only part of the system that is working - sorta.

This bring back memories of the outage about 5-6 years ago when they were down 7-10 days

[lammert]

Five years ago a central router died at their data center and a replacement had to come from far. It could be a comparable problem now where they have to wait for equipment to be flown in.

[herb]

Some of the error messages they are displaying.

Forbidden

You don't have permission to access /widget#*$!X.htm on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, [e-mail] and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

Internet Explorer cannot display the webpage

[bobothecat]

I have one site that hasn't had any problems, yet another site (hosted on a different server) is down completely. At least I lucked out on one of them. Argh...

Thankfully the rest of my sites are hosted elsewhere.

[herb]

According to WestHost,

"ALL sites do not have full functionnality"

"The server that runs the backend for the hosting platform is currently down.

Even though your server may be displayed as up at: [netstatus.westhost.com...] the server that is down affects all of your server's backend processes. This will prevent your server from functioning properly".

WestHost Forum: [forums.westhost.com...]

[lammert]

the server that is down affects all of your server's backend processes.

Seems like a single point of failure just as with the router 5 years ago. The only difference is that a router is merely hardware and can be replaced, but a server which serves backend processes must be reinstalled, configured and a restore of data must take place. Hopefully they have a good functioning backup procedure for such a critical item. Just thinking which processes are running on such a single server which causes ALL sites in the data center to malfunction. Keeping fingers crossed...

[CernyM]

Last night they told me that the server that provides the virtualization for the accounts was down and had to be rebuilt.

I asked them why westhost.com worked and they claimed that it had been affected as well but that they were able to get it up and running again.

No good answer as to why they couldn't do the same thing to make my server work again.

[CernyM]

Status on their page now indicates that a fire suppression system was inadvertently triggered during an inspection.

[herb]

The real reason comes out...

On Saturday, February 20 2010, at approximately 2:20 p.m. MST during an annual fire system inspection of our data center the fire suppression system was inadvertently triggered and discharged into the data center environment. The discharge of the fire system has affected the hardware and operations of our servers. Not all servers have been affected by the discharge. We are working to restore our servers and services as quickly as possible. Due to the nature of this interruption we will be experiencing extended outages.

[rise2it]

So much for hope....they've now moved their estimate from 8am to 8pm mountain time, and it looks as if nothing has changed in the last 3 or 4 hours.

Personally, I grabbed some different hosting, spent a few hours during the night uploading files there, and changed my DNS entries. I've already gotten a few orders today (although I'm sure I'm losing a lot since it takes up to hours for these changes to make it through the internet, but at least I'm not completely dead.

You other guys may want to consider doing the same, because it appears Westhost is not going to be honest about the problem or realistic about how long it's going to take them to fix it.

[edited by: rise2it at 7:01 pm (utc) on Feb 21, 2010]

[BillyS]

I wonder if they have any kind of insurance for this type of event. Very unfortunate situation.

[CernyM]

You guys may want to consider doing the same.

I wish it were that easy for us.

We should be back up in some form tonight, with or without westhost, but it has been a major heartache.

[Nativeozzie]

We're out of business until they're back up. I've been on live help through 4 or 5 ETAS and said to their customer service you know very well after the 8am ETA they'll add 12 hours to it. He disagreed and said he has faith in their system admins.. I guess I was right because they added 12 hours to it.

Westhost has the only recent copy of our database, so we're just dead in the water. It's monday morning here for me and we can't process any sales, we're dead. We can't do a thing.

[rise2it]

"On Saturday, February 20 2010, at approximately 2:20 p.m. MST during an annual fire system inspection of our data center the fire suppression system was inadvertently triggered and discharged into the data center environment. The discharge of the fire system has affected the hardware and operations of our servers. Not all servers have been affected by the discharge. We are working to restore our servers and services as quickly as possible. Due to the nature of this interruption we will be experiencing extended outages."

[rise2it]

Nice... for them to admit the TRUTH 24 hours later.

Had that told us that when we began having problems 23 hours ago, we could have taken proper steps THEN knowing this was NOT a problem they were going to be able to solve in 1 hour, 3 hours, 3 more hours, etc.

... and THEY knew it, too.

[edited by: rise2it at 7:46 pm (utc) on Feb 21, 2010]

[Nativeozzie]

Yes, I've read the canned responses too. Their ETAs are now completely meaningless.

[Nativeozzie]

Oh the odd thing, I had another site down with Westhost, it was a phpbb based forum, social, not business based. It was showing "forbidden" to access on front page. It came back up and is functioning. So why did that site come up and not the other? I don't get it.

[rise2it]

the forbidden (403) is the error mine was showing, also.

Looks like only about 5% of their stuff went down, and rest was unaffected.

I'll trust it's okay to post this link here (for their status)

[netstatus.westhost.com...]

If had I had to guess, my guess would be they simply don't have enough extra equipment to replace what was damaged, much like the host that had the datacenter explosion a few years back.

All of which is understandable, but I'm disappointed in they way they chose to handle it. It's the weekend, and if this had happened an hour later, I would not have even known about it until tonight or early in the morning.

[edited by: rise2it at 8:17 pm (utc) on Feb 21, 2010]

[CernyM]

Looks like only about 5% of their stuff went down, and rest was unaffected.

Where are you getting that from? Their support folks told me 90%. The status page shows servers being up that aren't actually up. My server, for example, has been ping-able throughout the issue, but you can't actually get to it.

[rise2it]

"Where are you getting that from? Their support folks told me 90%. The status page shows servers being up that aren't actually up."

------

My 5% guesstimate was from the little red dots on the status page.

...based on what you said, they aren't being truthful about that, either...if it's really 90%, it's going to be DAYS.