| This 45 message thread spans 2 pages: 45 (  2 ) > > || |
|WebHostingTalk Hacked and Offline|
Worst incident in ages
WebHostingTalk [webhostingtalk.com] was maliciously attacked over the weekend. WebHostingTalk is the largest online forum for discussion of Webhosting and Server related issues. WebHostingTalk is owned by iNet Interactive [inetinteractive.com]. They are also owners of HotScripts.com, and Search Marketing Standard magazine. They also own numerous other forum sites. These guys are not newbies to forum operations and have a quality tech and management system in place.
A hacker gained access to an offsite backup server and then used info on that server to walk into the main live server. The hacker deleted the backup databases, and then deleted the live site. Apparently, they also covered their tracks and over wrote the drives so that no possibility of recovery was possible. This is the most deliberate, sophisticated and calculated hack I have heard of in recent memory.
Unfortunately, the last local offline copy of the system is from late last year. So expect them to be offline for a bit, while they rebuild the db's.
This is a lesson for ALL forum operators. Our thoughts are with the WHT and iNet teams that are working on the issues.
/off doing backups to dvd disks.
Interviews from HostingCon2008
Including interview with iNet CEO Troy Augustine [searchengineworld.com]
Also, a previous thread on the topic here: [webmasterworld.com...]
[edited by: Brett_Tabke at 8:01 pm (utc) on Mar. 26, 2009]
User should also know that the WHT passwords are supposed to be going around as well.
|doing backups to dvd disks. |
Nothing wrong with that but I would only use it as "A" backup and not "The" backup. There is few things to keep in mind about writable DVD's, first and foremost be sure to use quality media. The "bargain" discs can have failure rates as much as 50%. Do not fill the disc to capacity, leave about half a gigabyte empty. Corruption when it happens is more likely to occur at the end of a burn. Lastly you're burning into a chemical that will eventually break down over time leaving the data unrecoverable. This can happen in a fairly short time or possibly take a very long time, people have had discs go bad in as short a time as 1 year. Your DVD burner, the burning software or even the firmware can also play a role in how well writable discs are written.
The bottom line is don't store anything on DVD unless it's expendable.
2 me it looks like an inside job (or use to be inside) to get back at the company for one reason or another as this really isn't normal behavior for a hacker but the behavior of someone wanting to settle a score.
nice place to hide a log in.
|These guys are not newbies to forum operations and have a quality tech and management system in place. |
|Unfortunately, the last local offline copy of the system is from late last year. |
Am I the only one questioning the first quotation?
|this really isn't normal behavior for a hacker but the behavior of someone wanting to settle a score. |
i could see a technological rendition of payback on film where that midget from jackass plays the hacker.
|doing backups to dvd disks. |
what's wrong with external hard drives?
coopster I saw that myself and thought what the heck and they are the leader in webhosting but don't make a backup for themselves. I wonder how many post "were" in their forum on that topic.
|2 me it looks like an inside job (or use to be inside) to get back at the company for one reason or another as this really isn't normal behavior for a hacker but the behavior of someone wanting to settle a score. |
phpbb.com just went through a similar situation and all indications are in the words of the hacker "because I could". The exploit he used through a third party script did not require a lot of knowledge to do severe damage. It's very possible someone with very little knowledge did this, e.g. immature kid with nothing better to do.
Hey hacker, yes you, when you figured out you weren't smart enough to build your own quality forum and decided to hack WHT you broke the law and the law always catches up to you. Not very bright...
Well, if I were them, I'd be on Google right now running a script to download a cached copy of every page and inserting the missing threads back into the DB.
A tedious task, but worthwhile.
Coop, those are my words - not theirs. Umm, I doubt there is 5% of the top 10000 sites that go to the expense of having a full system syncing backup online at a shadow server setting behind a tight firewall. That's pretty extreme and very secure. So much so, it clearly lead them to be over confident in their offline backup procedures.
Ya, I have several backups online and here on disk. The problem is keeping them all current but the most recent one. Hence - I try to backup to dvd atleast once a month, but realistically it drags to once a quarter.
ugh - sorry card_demon. we'll just cross link between the two threads. Stitching them (because this is a homepager) would be problematic.
> Well, if I were them, I'd be on Google right now
> running a script to download a cached copy
that is an option and one they are exploring. However, they don't want to get banned for spidering Google without permission.
|I have several backups online and here on disk |
I even have a minimal copy of my main site encrypted on a memory stick on my keychain so worse case if something catastrophic were to happen I can bootstrap my site back online from anywhere.
Of course I would lose all the correspondence, screen shots, and the data would be a few days old but that's better than a total loss as you can always start making thousands of new screen shots and anyone that didn't hear from you can write back!
Besides that, it's on a server, a backup server, 2 local PCs (rotating copies) and a couple of backup drives... maybe I should burn a new DVD too :)
Seriously though, I don't care how hardened your firewalls are the least path of resistance is actually to gain access to someone's desktop machines and tunnel back into the servers from there.
Not that I've ever done this, but I had it happen to a customer once, and I was pulling my hair out cause I knew that server was locked down tight as a drum which actually pointed back to their PCs!
Hope WHT gets back online and figures out what went wrong.
[edited by: incrediBILL at 8:50 pm (utc) on Mar. 26, 2009]
Gotta admit, hitting the backup to get data required to hit the host was pretty good..
The act of deleting everything is quite brutal, well well well beyond a defacement, I hope it doesn't turn out to be some script kiddie that did it..
I have gone through some long threads over there. Looks like they have lost some 6 months worth of forum posts.
Apparently they have more recent backups on DVDs but those backups are not good or may be not complete. It appears that they never attempted to restore the complete site from backups before this incident. So they never verified the integrity of their backups. When the backups were eventually called for, they failed.
This is like a situation where fire extinguishers refuse to work when the building is on fire.
Moral of the story is, not only make multiple backups but once in a while verify those backups as well. Try to restore those backups on test domains/servers and make sure that they work.
Most people are in too big a hurry and don't use the "verify read after write" option which quite easily validates that the backup on the media is at least good.
One gotcha though, once I was having a hard disk die and tried to make a backup of the dying drive and it was having sectors drop out all over the place and the backup software didn't report the HD errors!
So restoring and verifying is about the only way to be 100% sure you have a solid baseline backup.
|Coop, those are my words - not theirs. |
After I posted I realized I said "quotation" and it may have been read differently than intended. Perhaps if I phrased the question as "Am I the only one questioning the quality tech and management system in place?"
Even if they are your words or if they were not said at all, I would still question the (lack of) disaster recovery plan.
|I doubt there is 5% of the top 10000 sites that go to the expense of having a full system syncing backup online at a shadow server setting behind a tight firewall. That's pretty extreme and very secure. |
I think your stats may actually be a bit liberal. However, I would say liberal in regards to a solid disaster recovery plan in general as opposed to the synch/shadow reference. Disaster recovery takes work and is often dismissed as "it isn't going to happen to me". Well guess what? It may indeed happen to you and if you rely on the operation for your livelihood you may want to reconsider your thought process. It doesn't have to be that extensive. Yes, it could be. But plan, budget, develop, test and implement. And for goodness sake, develop a continuous improvement process, even if it is merely an analysis meeting once a year to be certain your plan is solid.
Note, the "you" reference above is not directed at anybody in particular. It means me, you, anybody reading, that is concerned that something like this could happen.
Perhaps it's just me and the way I operate, but disaster recovery is part of my planning from day one. I learned it early on but really applied my experience during the Y2K hype where it was tested and proven.
|So much so, it clearly lead them to be over confident in their offline backup procedures. |
I think that says a lot right there. And ...
|Moral of the story is, not only make multiple backups but once in a while verify those backups as well. Try to restore those backups on test domains/servers and make sure that they work. |
What more can you say?
Concerned about disaster? Have a recovery plan? If you haven't struggled with these questions my advice is don't wait until it happens, you'll be quite upset with yourself when it does happen.
The best thing about the news is that it happened to somebody else and brought your awareness level to the point of recognition. Man, do I sympathize with the folks over at WHT. The situation is miserable for them and we know it. But I guarantee you that the first thing they would tell you would be to develop a better plan.
*** Unfortunately, the last local offline copy of the system is from late last year. ***
Ouch. That is very very careless.
wow, I just signed up for them. Didn't realize they were so large. Surprised they didn't have a more recent backup considering the size. I guess this means I lost all my posts. Oh well.
[edited by: CWebguy at 11:31 pm (utc) on Mar. 26, 2009]
>>>>>Ya, I have several backups online and here on disk. The probably is keeping them all current but the most recent one.
This is what I do (kinda plug to a software that I create, without naming it - but it works).
1. On my dedicated servers, I have scheduled tasks to export the databases to specific folders hourly & daily.
2. I then rsync and robocopy (depending on OS) all required folders, including db_dump, scripts, logs etc to a "tobackup/yyyymm" folder. Yes, this requires a little over 2 times of the local space on the dedicated server, but larger disks on servers are cheap (comapared to bandwidth).
3. Then I upload the "tobackup" folder to Amazon S3. Robocopy (or rsync) and upload to Amazon S3 is done in a single schedule job. This way, I have monthly backups available at S3, which are updated every morning. Once in 4-5 months, I login to my S3 account and delete older backups.
4. To backup the backup, I have another scheduled job, which runs on my home machine, and downloads the files from Amazon S3 to an external hard drive and then robocpy's that drive with another similar sized drive. This way, I have data on two local disks at home for past several months, and at Amazon S3 for past few months.
Very inexpensive, and it works. I have 1 day old data, available on a high speed network, which can be restored very fast.
I lost a lot of data once, I'm now fanatical about backing things up. I've got backups hidden on the server itself, I've got local copies that are backed up elsewhere on the fly.
Learn from someone's pain. Backing up your websites is what being a webmaster is all about.
Wow. Scary stuff.
I keep local backups about once a month beyond my off-site automated back-ups. Maybe I should do more. But man... that is malicious.
The WHT password/email/username database has begun showing up on various file sharing sites. If you were a member, best change your passwords immediately if you used them anywhere else. The hashed passwords may be salt'ed but with such a vulnerable target I wouldn't be surprised to see someone find a way to crack the hash.
As more details emerge I share the sentiment that the owners were extremely careless and the members are paying for it, if you spot a copy of a database some 5000+ pages long WHT is asking to be notified.
I have offsite backup for my dedicated server, but I don't rely just on that.
I have a machine that runs 24 / 7 solely to backup my websites. I use Navicat to backup the db's (every night on my forums) and WSFTP to run scheduled FTP downloads.
Then that machine is backed up at home (on external HD) and offline (using a service). Not to mention I keep a backup hosting account configured for my websites, and I use managed DNS so I could literally switch over to that in a matter of minutes.
Sounds like iNet has a thing or two to learn about redundency. 6 months worth of posts is a lot of data, and that's a lot of user info to get compromised.
I use a tool call rsnapshot. It pulls backup from my web server and stores in weekly, hourly, daily format. Same tool run from my Mac book and mac book uses Time capsule to make backup. My ISP also provides NAS backup account.
|User should also know that the WHT passwords are supposed to be going around as well. |
Wondering, even if the credit card details were stored in backups !
Looks like they have a data loss from 10-14-2008 to 03-23-2009 more then 150 days
[edited by: Future at 9:42 am (utc) on Mar. 27, 2009]
I've talked to a few people "in the industry". Before revealing this problem, I posed the following question:
"I have a website. The server is locked down, and is RAID protected. It is synched to off-site backup which is also locked down, with RAID protection. Would you consider this sufficiently secure?"
Most said yes, although most were hobbyists.
|The server is locked down |
What do we actually mean by "locked down"?
IMHO that's a nice fuzzy phrase bandied about which has no precise meaning. Ask three people to "lock a server down" and you'll get three different results :-)
|It is synched to off-site backup |
Continuously synced (i.e. replication)?
Synced on a schedule (every X days/hours)?
Synced manually (every time the sysadmin feels like it)?
Q: If you get data corruption on the 'master' server does that corruption get synced to the backup server automatically? Whoops.
Q: Does your master server have logon credentials/keys for the backup server stored on it? If so, master server gets hacked - hackers can log in to the backup server straight away. Whoops.
Q: Does the backup server have logon credentials/keys for the master server stored on it? If so, backup server gets hacked - hackers can log in to the master server straight away. Double-whoops.
Ok, a bit of precision, and a question.
A friend from Uni has a proper multi-million grossing site with no offline backup. He says he doesn't need it. What do you think?
Application server is only point-of-exposure to the internet. It has no authority to do anything to any other bit of hardware. Configuation cannot be done over public connection. Watchguard or similar is DNS target, forwarding HTTP or HTTPS traffic to App server, bloking all other ports and otherwise protecting against malware and attack. SQL-injection is prevented. Application server actually has mutiple copies offline for development purposes.
App server has access to the DB server. DB server can ONLY be accessed from App server, plus managment server, not from internet.
Management server can only be accessed over VPN. VPN can only be established from pre-determined IPs and terminates on watchguard, then a seperate SSL VPN needs to be established through that to management server. (Man server runs the backend stuff, including CRM). App server is on a separate VLAN to Man server.
Offsite backup is on another HW firewall, only accessible by VPN, which again is limited to a fixed set of IPs. Backup is daily. VPN is established by Man server to backup site. Backup site is for DB only, and does not have access details for main site. Backup site is NOT a pre-approved VPN client.
Doesn't sound too dissimilar to WHT, appart from the VPN lock-down, does it?
> Wondering, even if the credit card
> details were stored in backups
No. They have specifically said they do not keep them.
> Looks like they have a data loss from
> 10-14-2008 to 03-23-2009 more then 150 days
They have the data from sooner, but apparently there is trouble with the data and getting it restored.
> It is synched to off-site backup which is also locked down
That is the question. From reading the long thread over there, it is pretty clear that only those working with the backups actually even knew what the server address was - let alone how someone got into it.
| This 45 message thread spans 2 pages: 45 (  2 ) > > |