homepage Welcome to WebmasterWorld Guest from 54.234.217.88
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
Forum Library, Charter, Moderators: phranque

Website Technology Issues Forum

This 129 message thread spans 5 pages: < < 129 ( 1 [2] 3 4 5 > >     
Westhost sites down
rise2it




msg:4084071
 11:24 pm on Feb 20, 2010 (gmt 0)

It appears anything hosted by Westhost has been down since about 3:30pm eastern time (roughly three hours now), and they are working on the problem.

 

bobothecat




msg:4084410
 8:20 pm on Feb 21, 2010 (gmt 0)

My server, for example, has been ping-able throughout the issue, but you can't actually get to it.


It's really strange. My one site that is affected mostly shows either 404's or 403's, but I'm still able to retrieve some pages with no problem... oddest thing I've ever seen.

Nativeozzie




msg:4084412
 8:39 pm on Feb 21, 2010 (gmt 0)

I actually registered on Webmasterworld to find a new host, I was only with Westhost because years ago webmasterworld had a link to them as their host.. now I find you don't seem to be able to find that sort of info here.

willybfriendly




msg:4084433
 9:48 pm on Feb 21, 2010 (gmt 0)

It seems they are making progress. Had two sites that were apparently unaffected. Another (e-commerce) was back on line this morning, but I had to repair a couple of DB tables (most likely due to visitors being onsite when everything crashed). A third has come back on line in the past couple of hours.

I can be pretty forgiving of hardware failures. S**t happens, and I don't expect them to have a semi truck load of redundant equipment on site. Shucks, I have seen a major State govt agency lose their system for two days over a single part that had to be shipped in.

Lack of transparence when things go wrong is troublesome.

Nativeozzie




msg:4084436
 9:55 pm on Feb 21, 2010 (gmt 0)

They have now added the following to their outage info..


"The network service status listed below is inaccurate due to hardware failures."

rise2it




msg:4084476
 11:11 pm on Feb 21, 2010 (gmt 0)

They have now added the following to their outage info..


"The network service status listed below is inaccurate due to hardware failures."

---------------------

Maybe one of them is reading this thread :-)

Again, I think they should have posted that YESTERDAY.

The problem isn't that they had a situation....the problem is in how poorly they have handled the situation.

It's good to hear thoughthat someone's site has come back to life.

bobothecat




msg:4084478
 11:24 pm on Feb 21, 2010 (gmt 0)

"The network service status listed below is inaccurate due to hardware failures."


They should have the same disclaimer for the "Fix ETA".

If you want my personal opinion... I think they were hacked.

Yesterday evening when the outage occured, I noticed that my SecureCRT connection disconnected... after trying to re-connect, it said the password was invalid, asking for me to type it in again (which I didn't), that's when I knew something was wrong and called.

Thus far the server is still reachable by ping and traceroute... kind of odd since it's supposed to be 'offline'.

Some page requests still come up, while others don't (as mentioned earlier).

I'm posting this now, only because after spending nearly 24 hours talking to 'technical support' asking to speak to an actual tech. I've been refused.

[edited by: bobothecat at 12:20 am (utc) on Feb 22, 2010]

rise2it




msg:4084480
 11:37 pm on Feb 21, 2010 (gmt 0)

Going through the Westhost forum (on their site), it seems everyone that calls in is being told (and has been told) something different.

simplesimon




msg:4084518
 1:51 am on Feb 22, 2010 (gmt 0)

They added 12 hrs to the eta again - back to 8AM (tomorrow now). Already moved one website today. Guess I'll be moving the rest. I have been stalling on moving my email off of their servers. I just hope their backups are intact so I can get the 2 days worth of email.

Lorel




msg:4084519
 1:52 am on Feb 22, 2010 (gmt 0)

My site has been down for over 24 hours except for a brief respite when I got in some email this morning. I have 35 clients on Westhost, mostly on different servers and some of those sites are up- although their email/carts/etc may not be working.

I'm not about to switch from Westhost however as it's the best host I've every used. They are constantly updating their Software/hardware so I'll just trust they get this worked out quickly.

Nativeozzie




msg:4084520
 1:53 am on Feb 22, 2010 (gmt 0)

I think it's long past getting worked out quickly don't you?

rise2it




msg:4084526
 2:31 am on Feb 22, 2010 (gmt 0)

Well, if the reason was the tester from the company that installed/maintained the equipment that caused the accident and set the ball rolling, then imagine the damage/stress caused to the Westhost people.

Again, though - with things this far out of control, they should have notified everyone THEN so we could have started making other arrangements, because they had to know THEN things were not going to be fixed in three hours.

I did trust my gut and switched early this morning, but the 12 hours I waited because they chose not to be honest about how bad things were cost me (just as it has cost a lot of you) a lot of business.

Now, I'm not the most tech savvy guy on here, but now that we're looking at 36+ hours by in the morning, is it not possible for them to somehow stick up a page on the affected domains so visitors aren't getting 403/404 errors and thinking we are no longer in business?

What are returning/repeat customers going to think? What is the customer that had a problem with an order and tried to contact us over the weekend going to think?

Their customer's first assumption is going to be that WE have gone under.

Again, this is where Westhost needed to STEP UP.

lammert




msg:4084555
 3:54 am on Feb 22, 2010 (gmt 0)

My experience is that technicians are optimists by nature and they just hadn't any idea of the size of the problems at the beginning. This looks like a Rackspace problem a few months ago where a transition from normal to UPS power caused havoc. Instead of just restarting the servers, they had to replace a few hundred power supplies, routers, switches and other networking equipment before everything was running again.

An estimation of the size of the damage and the needed action and the time frame for repair is often subject to iteration in this kind of situations. But I agree that they should be able to make a good estimation of the ETA now and communicate it to their customers.

Nativeozzie




msg:4084569
 4:49 am on Feb 22, 2010 (gmt 0)

I have zero doubt there will be a further ETA added when it starts getting close to 8am, currently have lost 2 full days trading and are unable to ship out previous orders or communicate with customers at all.

CernyM




msg:4084587
 5:56 am on Feb 22, 2010 (gmt 0)

Ours finally came back up and I've heard others have come back up through the day. Looks like its just a long slog.

Nativeozzie




msg:4084598
 6:25 am on Feb 22, 2010 (gmt 0)

Ugh, why the heck can't mine come back up.

oddsod




msg:4084780
 2:36 pm on Feb 22, 2010 (gmt 0)

This a good example of how not to handle things: lie to your customers, keep changing the ETA, don't answer any questions, don't provide any information...and when you do post information (example thread [forums.westhost.com]) make sure it's virutally useless corporate speak: "We value your custom and are working hard blah blah."

All my accounts with them are still down, including multiple dedicated servers. Today is the third day. And no idea when they'll be working again.

I have zero doubt there will be a further ETA added when it starts getting close to 8am, currently have lost 2 full days trading and are unable to ship out previous orders or communicate with customers at all.

Well that's if they can even be bothered to update the status page any more. It's a few minutes away from that ETA right now and there's not the smallest glimmer that my sites are back online.

cmendla




msg:4084783
 2:49 pm on Feb 22, 2010 (gmt 0)

I guess it could be worse. The problem I have is that the email I use for almost all of my accounts in on a site that is currently down due to the WH thing.

I'm hoping that it will come back up some time this morning.

bobothecat




msg:4084790
 2:55 pm on Feb 22, 2010 (gmt 0)

It's a few minutes away from that ETA right now and there's not the smallest glimmer that my sites are back online.


I'm sure they'll just change it to 8:00 PM... and continue with the same old babble as usual.

simplesimon




msg:4084794
 2:57 pm on Feb 22, 2010 (gmt 0)

If there are just a handful of sites left I doubt they are going to sweat it. About a month ago I had an IP stop responding during launch on Friday night and the "techs" on weekend duty had no clue. Nothing was done about it until the admins came in on monday morning, and when they finally fixed it they didn't bother to tell me. Of course I had launched elsewhere by that time. Westhost sure gives you good practice in moving and restoring websites ;)

oddsod




msg:4084801
 3:16 pm on Feb 22, 2010 (gmt 0)

Tell me about it!

Apparently, and according to tech support, the backup servers are in the same datacenter!

That's not what they told me when they sold me the service of nightly backups for dedicated servers. They sold it on the merits that as it was in a different location it was a lot safer.

ETA has passed and the latest news is that they don't have a new ETA.

Demaestro




msg:4084829
 4:18 pm on Feb 22, 2010 (gmt 0)

All my sites are up from them.

One of my email servers is still down. At least my sites are up, I didn't see any interruption to any of my sites, but I still don't have my top level email.

Totally unrelated to this issue I have been planning a move from them in the next couple months.

I may speed up my plans though and get everything I have moved out of there sooner than I wanted to. My reason was a recent price change in the wrong direction and a very good local quote, but reading this thread and a weekend without email has made me rethink my time line.

A few years ago they were one of the only places willing to let me have mod_python on a non-dedicated server and for that I loved them, unfortunately the shine has worn off.

[edited by: Demaestro at 4:21 pm (utc) on Feb 22, 2010]

oddsod




msg:4084830
 4:18 pm on Feb 22, 2010 (gmt 0)

Even better, they claim that 98% of their customers are back online (up from 95% earlier).

They've now got 75%(?) of customers all bemoaning the fact that they're in (what they think is) the non-working 2%.

justgowithit




msg:4084849
 4:57 pm on Feb 22, 2010 (gmt 0)

My sites were all back online yesterday. I've got sites hosted on several different servers with them. Some were unaffected, others weren't so lucky.

blend27




msg:4084864
 5:20 pm on Feb 22, 2010 (gmt 0)

-- weren't so lucky --

Why?

simplesimon




msg:4084865
 5:22 pm on Feb 22, 2010 (gmt 0)

I have seven or eight clients with their own westhost accounts, and my own reseller account, and none of those were affected. The one account that is still down has certainly kept me busy through the weekend though.

incrediBILL




msg:4084885
 5:55 pm on Feb 22, 2010 (gmt 0)

Having been a host in a past life you should cut them some slack because something like a fire suppression system going off is a really big mess.

People complaining they're "out of business" today only have themselves to blame for not having a backup server or some other contingency plan.

What if Westhost were hit by a tornado or a flood or some other natural disaster?

You need to be prepared for the inevitable, not Westhost, as it's your business.

I have servers in 2 different data centers and a clone at home, so unless some supreme deity with a sense of humor knocks out both data centers in extremely opposite geographical locations, I'm prepared, are you?

oddsod




msg:4084887
 6:03 pm on Feb 22, 2010 (gmt 0)

I'm sure they'll just change it to 8:00 PM... and continue with the same old babble as usual.

They did change it to 8 PM.

Now they've changed ETA to "we don't have a clue". (more precisely: Fix ETA: TBD)

I have servers in 2 different data centers and a clone at home

Well done. We can't all do that or want to do that. I pay Westhost for a dedicated server. I pay extra for their nighly remote backup service. I decided that that was a sufficient level of backup for what this site was worth to me. Problems happen, but it's how you deal with it that upsets people.

They screwed Brett Tabke around about five years ago (I take it he didn't have a spare copy of his site under his bed).

rise2it




msg:4084936
 7:21 pm on Feb 22, 2010 (gmt 0)

The choice is up to you, but I chose on late Sat/early Sunday to not trust this would be a quick fix, and pointed the DNS records to a backup server.

While we've still lost a LOT of business (because of the time it takes the Domain Name records to update), we're not dead in the water. (Luckily, we don't have our online databases there, as some of you did...so it's basically just webpages and email).

For those sitting on the fence, I'd recommend grabbing some cheap hosting elsewhere, and uploading your files (at at least one static page for your customers to see even if you for some reason do not have your own backup files). Once you change the DNS entries, you'll start to get some business trickling in within 3 or 4 hours.

Once Westhost gets this fixed, you can then point your stuff back to their servers.

My BIGGEST gripe with them isn't that the site is down, but that the results they are serving are still 403/404....surely they could have at least routed something for our affected domains so at least the customer would see a webpage letting them know there was a tech issue, instead of the customer seeing nothing and thinking we were no longer in business - and I have spent today fielding THOSE types of calls.

There have been a couple of comments on other forums that some of the sites have DID get restored are now down AGAIN, and unfortunately I think this situation will continue to play out this way over the next few days, and maybe longer, as they swap hardware, restore backups, etc.

Demaestro




msg:4084965
 8:01 pm on Feb 22, 2010 (gmt 0)

bill you raise excellent points.

What is really crappy though is that most of the people who have an account with Westhost have a backup plan with westhost.

I have backup files of all my sites locally so I can get any site back up all I need to do is wait for DNS to resolve.

What about for my IMAP email though? They claim to do an offsite backup? Why isn't that offsite backup online?

Is there anything I can do in the future to give me more control over my IMAP email so that if something like this happens that I am not missing 2 days worth of emails?

lexipixel




msg:4085003
 8:40 pm on Feb 22, 2010 (gmt 0)

I've got one WestHost hosted domain left that still isn't coming up, (web or email), the other 60+ sites have all come back.

Demaestro




msg:4085007
 8:44 pm on Feb 22, 2010 (gmt 0)

The error message from my email program has recently changed from:

"Can't find address"

to:

"Connection is refused"

This 129 message thread spans 5 pages: < < 129 ( 1 [2] 3 4 5 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Hardware and OS Related Technologies / Website Technology Issues
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved