Westhost down again

Fiber line was cut by construction company

[justgowithit]

All of my sites with Westhost are DOA. They've been down going on about 10 minutes now. Even their site is down.

We're in the middle of an upgrade, then it's bye-bye to Westhost.

[ypsites]

yours still down? i have several WH sites that are down ....

[samc]

the whole network is down :(

[keukalake]

The last time this happened they claimed that the sprinklers went off unexpectedly in their data center. They had to rebuild servers and sites were down for a couple of days.

[plestus]

This is bad. Really bad. 18 clients calling every five minutes, West Host not EVEN A SINGLE STATUS update, an automatic message...nothing, just a guy sayong "We don't know, we can't tell you, we have no updates" Really? I give them 24 hours to get their act together or I am migrating everyone to the cloud.

[justgowithit]

They claim that a contractor cut their fiber optic cable. They seem to higher pretty poor contractors. First the fire system was triggered accidentally by a contractor, and now their connection has been cut.

[justgowithit]

They are updating their Twitter feed [twitter.com].

[keukalake]

I thought they had multiple bandwidth suppliers. What a disaster.

[Demaestro]

I left them after this happened the last time.

I am really sorry to hear it is happening again. I wondered about the validity of their last excuse and now I really wonder.

Last time their time estimates to be backup were horrible and unless you knew to go a specific forum you wouldn't have gotten any info.

Maybe it is time to cross "contractors fault" off the excuse list.

[willybfriendly]

Our sites appear to be back online. A bit after 2PM PST.

It does say something that we come to WebmasterWorld for information about such things...

[samc]

@willybfriendly yes and it reminds about something I need to take care of.. 'subscribe'

[jneeley]

As some of you know the downtime was caused by a construction company cutting the fiber line going into the data center. Although I don't know if XO has repaired the fiber yet all our sites are back online via rerouted traffic through another provider. Needless to say the servers were never down and were being monitored by our admins through an external connection.

We were very active on Twitter in giving timely updates and responses to all clients even though we didn't have an internet connection to our office. I wish I knew this conversation was happening earlier but I can assure you I'll be on Twitter @westhost.

I am concerned that anyone has considered leaving and will personally talk to you to resolve any concerns you may have. I can be reached by jneeley at westhost dot com if you want to talk... and please do!

-Jake Neeley, Brand Manager at WestHost

[smallcompany]

sites are back online via rerouted traffic through another provider

I would expect this to happen automatically when "main wire" goes off. I believe it's called fail-over. Even if some action is needed for fail-over to kick in, which translates into time, my sites were down (or unreachable) from roughly 11:54 AM to 2:13 PM PST which I recognize as tooooooo loooooong.

[jneeley]

@smallcompany the required action needed for fail-over to kick in was longer than usual and certainly to long. I don't know if you knew it or not but our site was down as well so when you mention this I can relate.

Our IT department was working fast and thoroughly after the initial line was cut to find any issues for the fail-over line. They determined that our fail-line was functional but that our provider wasn't announcing one of our blocks of IP addresses to the Internet so, when XO went down, one of our blocks went missing so to speak. When discovered we contacted our provider, they added our IPs and your site came up again.

We never wish a couple hours of downtime upon anyone so we continue to develop more and more plans to prevent unforeseen external events from pulling us offline. As I mentioned before I am very willing to talk with you or anyone else.

[samc]

Yesterday they decided to re-assign IPs on a dedicated server without telling anyone. Judging by supports lack of response I bet they did it to everyone. IP address change=different upstream provider.

Run away. This hosting company has nothing but problems.
1.fire suppression wrecks data center
2.untold issues with clients dedicated server. Support nice but takes weeks to resolve.
3.cable gets cut from their upstream provider (WestHost does not own it's network)
4.Yesterday, network wide software update (what they told us) causes problems. For us it reassigned all IP addresses on our dedicated server. Only way to contact was via Twitter (Jake above)

Nice people, terrible network.

[wheel]

Yesterday they decided to re-assign IPs on a dedicated server without telling anyone

Holy crow, that's crazy. That would basically take me right offline.
My provider was offline once I think in all the years I've been with them. I know they are failover crazy and that they know what they're doing. But they also keep us up to date.

When they went offline (result of a contracter shutting down both their cooling system and their failover cooling system) they did stuff like send an email saying 'data center temps too high, on my way to center to investigate' followed by 'at data center, cooling system offline and backup offline, cycling both right now', 'cooling systems back online, temps returning to normal. And two days later a complete report detailing what they contractor had done and the steps they'd taken to ensure it didn't happen again.

So I read the emails and didn't get excited. The data center guy was working on the problem and I had an idea of what was going on and what it would take to get it going again.

If he'd skipped the emails, I'd have spent my day figuring out what was going on, and trying to reach him, all at very elevated excitement levels. Instead I didn't miss a beat.

[samc]

I'd have spent my day figuring out what was going on, and trying to reach him, all at very elevated excitement levels.

Yes that has been the schedule with WH. I'm certainly not alone ether.
The part I love is they usually ask why you did not use:
1.chat
2.call
3.email
4.submit a ticket

Choose any other one but the one you are using on the list (it's a weekend, I 1. don't want to chat,2. certainly don't want to talk and 3 and 4 take 24 to 48 hours to respond. And always a pop-up recommending you get on chat (while you are on chat with a message saying they are all jammed up.

Last week the client was blocked by Firewall during business hours (not sure this is even true) and when client called support told them our DNS was being moved. So then I get a call from the client. :)

I call WH and I'm told the firewall tripped on the client. Everyone is standing around and support is telling my client the DNS is being moved.

Now if this were a war story from many years but this has all transpired in 9 months time. Before I really did not get any chance to talk to support because everything was stable on the network. They used to own their server farm. Now it's all outsourced to their parent company that outsources to it's other entities.

[jneeley]

Glad you feel we're nice people but we all know that can only get us so far. Similarly opposite is like two of the most famous restaurants in the US, Carnegie Deli in New York and Durgin Park in Boston; infamously rude waiters but food so good no one can argue.

We know we make mistakes and have to create work arounds for the mistakes made by our partners; both situations are not desirable and have negative affects. Because our entire business model rests on the effective nature of our network and backend systems we spend a huge majority of our time working to enhance it. As a result we own our server farm (purchased, racked and supported by us, not outsourced) housed in a very good data center connected through redundant networks.

I can agree that notifications regarding outages are not up to par which is why we're creating fixes for that as well. I think we can also agree that through stress we become better. Almost all major corporations today emerged during a time of external and/or internal distress. Because of our experience we can conceptualize and deliver on a higher level which means better products and services for our clients.

I am confident of where we going, that we've become better and genuinely excited for what lies ahead.

[wheel]

. I think we can also agree that through stress we become better.

I don't host with you, but quite frankly, it's the job of hosting companies to have prepared and planned for this ahead of time, not via trial by fire. You should have tested plans and failsafes already in place so that this stuff doesn't happen. Most hosting companies do.

Did you really change people's IP address? I've never heard of a hosting company doing that.

[jneeley]

You should have tested plans and failsafes already in place so that this stuff doesn't happen.

Like the example you mentioned before, there are times when unpredictable things happen, these things are the fire that we learn from. However, the testing and fail-over strategies provide opportunities to overcome those much faster than if nothing was in place.

Yes, IP addresses were changed as a result of IP allocation limits. Once IPv6 is in place this won't be an issue :)

[samc]

@jneely
I'm not going to harp on my list. I have worked with the 'partners' as you call them so perhaps I have a different view of the topography of the network(s). Semantics

Communication is key and communication that does not eat up your supports time or the webmaster's time (paid for by the client). Twitter is a good idea but not very secure for network announcements. I would suggest a means of email notification outside of relying on the default email account which is for billing and ends up being a general communication method (when it's used, much improvement needed).
The cost savings are in support time. NetStatus looks like it's all the old servers. Although colored status icons are not a replacement for sending emails explaining exactly what is going on.

I'm not a big fan of stress. I've had enough of it and in nature it has three outcomes:
1.)strength
2.)weakness
3.)death

The saying goes:

"That which does not kill us makes us stronger".
Friedrich Nietzsche

[samc]

So in the never ending saga of WestHost now we go and bind one of our 6 IP addresses only to find it's assigned to another live site (dedicated server) on the WestHost Network. So they took our IP addresses, changed the live sites, and then left us what?
the contract says 6 and what do you get?
Support can't do anything and must wait for 'some other people' to look at it (the U2 administrators in GB)
I can't wait till the contract is up (couple of months), Rackspace here we come!

[StoutFiles]

Like the example you mentioned before, there are times when unpredictable things happen, these things are the fire that we learn from. However, the testing and fail-over strategies provide opportunities to overcome those much faster than if nothing was in place.

Haha, are you starting to get defensive now? Whether you choose to believe it or not, this sort of incompetence is not happening with other hosts. Uptime is people's livelihood, and you come back with "oopsies, these things happen sometimes!" That's comforting.

No doubt you will be losing a lot of customers over this. Nothing you could possibly say in this forum is going to change that.

[TheMadScientist]

And yet again a chunk of their system is down...

By FAR the WORST host I've ever been with. I've hosted with 'sh****' hosts before and not had the issues I've had with these guys over the last 30 days... Asked for a refund because they totally f***** up their move and there's NO WAY to undo the damage to the reputation of the sites I host when they put an 'account suspended' message on all sites during the move, rather than 'moving, be back soon'. They didn't even have the courtesy to reply.

Absolutely the poorest system and service I've experienced in my 8 years online

[TheMadScientist]

These guys just don't get it do they?

I just got a reply to a ticket I sent today and it basically said 'no information should be lost for the service you provide' on the site I asked about...

Okay, neato, cool, what about the impression people get when they visit the site and it's down?

Is there some way I can explain to them if they use the service no information will be lost when my hosting company can't get it together and keep my sites up? I don't know who they are. I can't just call them and explain my hosting company can't keep the sites online but 'everything is ok'. All they know is they went to the site, it was not available (at least they didn't put a suspended message on it this time) and they leave thinking 'unreliable' and they're right... That's what WestHost seems to be the definition of lately.

Completely and totally unreliable.

Give me an 'oops' once a year for an hour... Alright.

Give me an 'oops' and then send me an e-mail that says everything's alright like it doesn't effect the site and system's reputation in the eyes of EVERY SINGLE VISITOR who sees the site's not working like it's no big deal and it's totally cool if they are down for hours on end (twice this month) and it's time for me to move, rapidly.

Can you image this:
We've used over half (I've been down for 5+ hours this month) our 99.9% uptime guarantee this month alone, but it's no big deal, there was no information lost, so all the people who saw the site was down will understand, even though you can't possibly explain anything to them, because you have no clue who they are.

Really? As a business owner I'm supposed to accept that from a hosting company... Huh, WTF?

ADDED: Of course their site's been up the whole time... lol ...I guess it's only important for sites to be up when it's their customers and their reputation on the line, not mine...

I'm honestly afraid to promote the site I just launched while it's hosted with them, because if I promote it and it keeps going down I can't undo the initial impression made by the site and system...

[TheMadScientist]

Sites slowed to a crawl again today... Started a chat... 17 minutes later without a reply from the representative I was supposedly 'now chatting with' I closed the window.