Webhost hack "wipes out 100,000 sites"

[encyclo]

Article [theregister.co.uk]:

A large internet service provider said data for as many as 100,000 websites was destroyed by attackers who targeted a zero-day vulnerability in a widely-used virtualization application.

Technicians at UK-based Vaserv.com were still scrambling to recover data on Monday evening UK time, more than 24 hours after unknown hackers were able to gain root access to the company's system, Rus Foster, the company's director told The Register. He said the attackers were able to penetrate his servers by exploiting a critical vulnerability in HyperVM, a virtualization application made by a company called LXLabs.

It appears the hackers got root access and did a

rm -rf

. The sites were on unmanaged, virtualized VPSes, so backups were in many cases the responsibility of the clients.

HyperVM is a web based management application than sits on top of Xen/Virtuozzo, not an actual virtualization application itself. The hack was a SQL injection via the web interface, known to the product developers but currently unpatched. HyperVM is used by many different VPS providers, who may also be vulnerable to a similar hack.

[tangor]

And I'm just getting my feet wet in virtualization... appears more STUDY is involved!

[driller41]

Obvious question is are there any other hosts at risk?

[mergen]

the obvious answer is that yes, there are many other hosts at risk. ;)

seriously though, there are many hosting companies that do not upgrade or update their software frequently enough in a timely manner. there is no way those guys will stay on top of something like this.

[tangor]

You start scratch from current (like me) and then nose to grindstone for updates/threats. That's all I meant in above. There's risk in everything these days. Some you have to accept, others you know, then you have to make the effort to get the skinny until you achieve trust in the product.

That's why they call it "work".

[jecasc]

Apparantly the owner of LXLabs, the maker of HyperVM has been found dead in his house. He hanged himself on monday.

[h-online.com...]

[JS_Harris]

There is more going on here than meets the eye but when hackers start leaving a body count I become more in favor of Obama and his Czar working with ISPs to secure their networks. This really sucked to read btw, pointless.

[BillyS]

Sad news all around.

[kartiksh]

yeah sad news all around. he seems hanged himself probably after hearing Vaserv and may be the referred canceled order (or lost order) could be of them due to this attack.

[swa66]

Details on how to attack lxadmin was made public on June 4th by an anonymous hacker. It actually contained a list of 24 ways described in detail on how to break the software.

I'm not going to link to malicious code, but the timeline the anonymous hacker left as a comment:

# Timeline :
#
# 05/21/2009 - sent initial email to vendor with a link to a private
#              resource for viewing various kloxo hiab575
#              vulnerability info
#
# 05/23/2009 - received the following: "Thanks for the info. I will
#              review this and let you know." (no signature)
#
# 05/30/2009 - sent an email asking if there were any updates
#
# 06/01/2009 - received the following: "Sorry for the delay. I am
#              currently looking into this, and will reply in a couple
#              of hours time." (no signature)
#
# 06/04/2009 - nothing heard from vendor, and the private resource
#              containing the vulnerability info still does not
#              appear to have been accessed

The apparent suicide of the author of course makes it even far more tragic.

Those affected by the outage can find status updates at
[66.71.245.2...] (I'd guess this is a temporary resource, but it;s been there quite a bit)

[johnnie]

Ouch! Makes one wonder about causality. Did he commit suicide before or after the attack?

[Hugene]

Sad news indeed. I think we need to understand that businesses and work are not the end of the world; as a matter of fact, nothing is more important than life itself.

[sgietz]

Ouch! I see at least one company going out of business here. This is bad news, but certainly no reason to kill oneself.

Very sad!

[mergen]

Wow, this is unbelievable... and incredibly sad.

[bwnbwn]

I read the article and what I don't understand is this response from the host. "I've heard from other people they've been hit by the same thing."

This exploit must have been around for some time and the hosting company just wasn't staying current with news as well as the Company that prouced the software.

All this could have been avoided with proper homework and staying up with current news assoicated with their business.

Very sad that this pushed a man to take his life.

From reading more into this it seems this has been a know exploit for some time and how to excute the exploit has been available and published on the web weeks before this happned.

Wehost should have taken protective measures. They had to have know of this issue and didn't take the steps to protect.

This could and most likely will lead to some legal action against them.

[encyclo]

It's certainly a sad and rather strange story. It's hard for the clients of the hosting company involved, but there is also the issue of other hosts using the same vulnerable HyperVM web interface. I doubt that Vaserv were the only users of this software.

It is rare that hacks of this nature are so destructive - often the aim of a hack is to introduce rootkits or backdoors and profit from the stored confidential information, rather than the chaos caused by a full delete.

It is also a reminder that virtualization and VPS solutions are not necessarily as secure as you'd think. It is possible for a hacker to exit the virtualized environment and access other instances or the machine's real root.

[incrediBILL]

This is a prime example of why I'm not a big fan of VPS sites because they're really not much safer than shared hosting.

Sad that my point had to be made by someone whacking entire servers.

This exploit must have been around for some time and the hosting company just wasn't staying current with news as well as the Company that prouced the software.

Read the timeline above, doesn't sound like it was known for very long.

Having been a host, we once had a RedHat vulnerability being actively exploited and there was nothing we could do to stop it until RedHat patched it, and they said it would take 7 days before the patch would be available. The most we could do was have a stack of clean drives ready and each time another server was breached we swapped the OS drive and it was back online, and that went on for 7 days.

Quite annoying but luckily nobody wiped the drives, they just installed rootkits and went away which is what the clean OS drive swap got rid of, over and over again.

[grelmar]

It is also a reminder that virtualization and VPS solutions are not necessarily as secure as you'd think. It is possible for a hacker to exit the virtualized environment and access other instances or the machine's real root.

A lot of that depends on how you configure the virtualized environment. And the same could be said about a great many web facing apps and utils (SQL anyone?)

The big problem is that virtualization is fairly new stuff. The people running it have little experience with it and there's a lot of new products on the market that haven't been thoroughly tested.

The power of virtualized environments is fantastic. You can clone VMs on the fly, distribute them across an array for load balancing. Depending on the technology and infrastructure you use, you can spin up new machines in seconds - compare that to how long it takes to integrate a new linux box into an existing cluster.

The downside...

There's a lot of security issues, both popularly known ones and fairly obscure ones.

And there are so many players competing for dollars now it isn't even funny. A few years ago, it was VMware or bust. Now... Sun, IBM, MS, all have virtualization products. And that's just the big names. Small virtualization outfits are a dime a dozen.

It's a market that's going to grow, and expect it to become a larger presence in the hosting sphere. But there's going to be some serious growing pains.

[MrHard]

Unfortunate, but you can switch hosts in a matter of minutes if you need to.

[jecasc]

Unfortunate, but you can switch hosts in a matter of minutes if you need to.

If you have a backup...

[Edwin]

Guess what, they don't have backups (even though that was supposed to be part of the managed service). At least, that's what I was told by support!

This was the extent of their message on the subject:

Hi
I'm afraid that there are no backups of this due to he hacker attack

Keep looking for the "sorry" all you like, it's not there...

[Gibble]

Never, EVER rely on someone else for backups. Unless you can see the backup, it doesn't exist.

[creeking]

quite a few of the customers were on the unmanaged plans, which specifically did NOT include backups.

[np2003]

This is one of those very tragic moments within the IT/hosting industry.

Hundreds of servers get hacked, and the software owner kills himself.

RIP and may these hackers ultimately face their karma.

[encyclo]

Guess what, they don't have backups (even though that was supposed to be part of the managed service).

I assume the backups were on similar servers which shared the same vulnerability. The sheer destructiveness of the attack probably wrecked the backups too.

[venetsian]

wow, that's pretty much a tragedy for this company. I know how it feels to loose client data. That's the main reason why EVERYBODY should invest into proper backups. The best is to use off server storage devices for situations where you have hacker attacks or failed hard drives.

[StoutFiles]

There's no excuse for not having your website backed up repeatedly. I understand there gets to be a point when it comes to large files and lots of them but it does not take long to store databases and most files on a local machine.

[venetsian]

Well, it depends on the costs involved. I suppose if they were offering super cheap prices with a lot of overselling then they may have it in the contract that backups are not performed. I know that there are a lot of fairly large web hosting companies that don't do backups as they consume a lot of IO and when the servers is too loaded it may crash it. We had some problems of this kind at the office. (Don't try to do backups when the load is high).

[ajaykumarmeher]

I am a vicitim of this incidence and for that reason My site was offline for more than 10 days. Lost all my data as had back up for only one of my site. I was so casual. Anyway I learned a lot from this incidence.