Hacked website - recovering from Google SEO perspective?

[realmaverick]

I need some urgent advice.

My biggest website, with over 2 million members, has just been hacked. To the nth degree. They somehow gained access to the root, they wiped the lot.

I'm still not clear how, but they then gained access to the backup servers and completely wiped them too. We're trying to find out if there's anything we can salvage.

The system was custom coded and was constantly developed, we've lost 10's if not hundreds of thousands of dollars of work and obviously the lost revenues.

I have a passionate team, who don't want to give up. I personally feel demoralised and don't really want to start again, as I can't even begin to imagine how.

If nothing can be salvaged and we did start over, what would be the best way to start over? We had hundreds of thousands of pages, my head is just completely fuzzy.

Any advice from an SEO perspective would be great.

Thanks a lot.

[Leosghost]

Wiped as in file tables or as in formatted? ( and if so how many passes do you estimate ) or wiped as in files deleted/trashed ?..

[slipkid]

Unfortunate that you do not have a backup of the website on some other media. I have copies of my site on a thumb drive, a cd, and my hard drive.

It is possible, unless blocked in .htaccess, that your site was indexed by the Wayback Machine. If so, that may be a source for most of your missing files.

[incrediBILL]

You might be able to recover pages from SE cache if you allowed them to be cached.

Nobody had a backup copy offline?

What about source control? Anything in SVN, git, etc.?

[realmaverick]

@Leosghost. My SA company said that he basically deleted all of the data, several terabytes of it. I don't know the finer details as of yet.

@Slipkid, the site was literally terabytes, if I could have backed up somehow to a hard drive offline, I would have. It was just too big.

@IncrediBill There are millions of cached pages, but the entire website was dynamic. Our users shared their art via our system, each page had a file that could be downloaded etc etc.

Nobody had an offline backup. I wish I'd have looked harder for a solution. But felt quite safe knowing we have backups on the main server AND on a remote server that I believed or was lead to believe was 100% secure.

They deleted everything regarding source control too. All that was left was an index.html, which they used to gloat about their "achievement".

The site receives tens of thousands of visitors a day, I just have no idea how or even if I can make use of that now.

[incrediBILL]

If you still have your membership list I would suggest that if you could restore the software and send out a request to people to upload the content again.

FYI, while doing offline terabyte backups aren't practical you could back up offline incrementally by only downloading what has changed on a daily basis. I do 'yesterday' backups to my local machine daily and can start restoring to a new server in a moments notice. Not that it will help the current situation but moving forward it can be done to avoid being so vulnerable in the future.

[edited by: incrediBILL at 1:04 am (utc) on Nov 1, 2012]

[martinibuster]

I know of a popular fishing forum that was hacked to bits. They lost everything up to three months prior, something like that but eventually lost that, too. So they started from scratch again. They're as popular as ever now, about a year or so later.

[realmaverick]

Thanks for the information. Obviously this is a huge blow, it still doesn't seem real. I'd been convinced this backup server was safe and secure. Oh dear.

I don't have a copy of the member base. We still have our Facebook page etc.

I just need to somehow funnel that traffic back.

It's not even as though I can reinstall a CMS or anything as the system was customised and intracate. If anybody has any advice on the best way forward. We have the domain. Starting from zero, got nothing to lose.

[Leosghost]

If they "deleted" it, that can ( and frequently does mean ) they merely told the drive to forget what it did with the data..ie which files are where etc..

This is also known as "low level" formatting..

This can be recovered from ..

But someone needs physical access to the drives that have been wiped ..either on the server or the drives need pulling and mounting as external drives to another machine..off it is the preferred way..

Either way, ( on original machine or off it ) the machine has to have an OS running on another drive ( not on the drive with the data that you want to recover )..


Then you need a "file recovery" program..which one depends on what the file system was that the wiped drive was using..and you need another empty drive , equal in size to, or larger than the "wiped" drive..
the file recovery software is set to read the wiped drive..this will take some considerable time ( depends on the size of the drive(s)..what it finds is then presented as "results" and usually as "a percentage of possibility of recovery"..or in other words how complete and faithful to the original, each file will be..

File in this case can mean html , jpeg, mpeg ,css, jscript etc..file recovery software will allow you to set what it is to look for..( so if you don't have tiff or 3D files , it wont waste time looking for them )..then when it has analysed the "wiped" you set it to "recover" to the empty drive..more than one wiped drive ? ..you need more than one empty one to recover too ..or at least as much space in data terms as you had in total on the wiped drive(s)..

This is basically the same process as is used to recover data from thumbdrives and camera cards..merely scaled up ..it can take a while if the drives you are recovering are over 1 terra..but it is not impossible even with multiple 2 terra drives..

Even if they have wiped by formatting..it is still possible..but the more passes they have made..the less possibility that you have of getting recognisable files back as the file integrity is compromised with each pass of a wipe program..But most vandals do not wipe by multiple pass formatting ..they merely delete and don't even run one full format pass..

Thus your entire site could be there still but merely have the "file allocation tables" ( or their equivalents ) wiped..and so the drive(s) will appear bare..even with a new "vandalised" index page loaded..

Do not write anything to the wiped drive at all if you want to try to recover the data..( because doing so would be writing over your underlying data..which may yet be recoverable ).best is to "pull" that drive..put in a temp redirect at your registrar to another account on another disc..and there ..with a page explaining what happened ..than do as I instructed above..

File recovery software is not horribly expensive ( some good ones at under $100.oo )..it is not a case of you get what you pay for..most is designed to recover from FAT or NTFS ..but some can also recover from UNIX ( linux or mac )..

Depends on what your server(s) was/ were running..

Sorry about any spelling errors in there ..I'm eating while posting ..but figured it was urgent..:)

You can also send the drive to a company specilising in data recovery ( if they don't need to open the drive then you won't need "clean room" services ) ..in which case it may not cost you more than a couple of hundred dollars plus the cost of a second drive for them to write to..

People are always wiping camera cards of weddings , baptisms etc..and occasionally they ( or someone wipes their entire HD )..

I actually a have a business which does data recovery..most is "I wiped my camera card" or "the kids wiped the camera".."clean room" work I farm out to trade houses..

But HDs that do not need opening ..

I have a machine and software set just to recover from HDs and "media"..

It isn't rocket surgery :)

And the software does the work.. while I do something else ..

Not looking for business, I never "tout" in this forum or elsewhere.. ( and I'm not on the same continent ;) ..but if you ask them ..then maybe you'll find that your host has an arrangement with a recovery company ..or a quick search will find you some ..maybe even within driving distance..

Or get the drive in your hands..download the software, and do as explained above..

[edited by: Leosghost at 1:39 am (utc) on Nov 1, 2012]

[slipkid]

@realmaverick

I apologize for my rather glib comment about a backup.

[incrediBILL]

I hadn't considered a file recovery program but that's a good idea.

Couple of issues I see is:

a) if they've replaced the index file and the server is creating server logs and/or processing email it's already busy obliterating files and should be turned off immediately.

b) if it's a rental server the host may not be willing to pull the drive and hand it over, certainly not without charging for the hardware.

c) the time involved before getting a site back online could be quite extensive

It's not even as though I can reinstall a CMS or anything as the system was customised and intracate.

I find it hard to believe that whoever did the customization didn't maintain a local copy.

[Sgt_Kickaxe]

This seems to be something that many webmasters will eventually face to some degree and all should prepare for, I feel for you because I know how much it stinks. You're doing what you can so... some other ideas:

Longshot: if your site was as large as it sounds contact Google/Bing/Yahoo to see if they can provide a static copy of the pages they knew about and get those back online, if they can provide them. I realize some functions will no longer work but it's best to maintain rankings for existing content if at all possible which won't happen if the pages remain down too long. Google/Bing/Yahoo is not in the business of providing such a service but on a larger site with solid content and thousands of users they *might* help.

Immediate: Quarantine all computers used by your team if it's in an office setting, obviously you can't do the same for home computers without court order. Someone is responsible and, unfortunately, the culprit is more likely than not to be someone in-house. Forensic evaluation might turn up clues but even if it doesn't you need to know, now.

Contact your CDN provider, if you had one, and ask about retrieving a backup copy of what they had.

Core code - this should absolutely have been backed up to an offline computer or drive as this is the heart and soul of any website. Your developers should each have had a copy or access to a dev server with a copy as well. The total size of the site may be terabytes but the core code would be much, much smaller. Contact your web developers and hope one of them still has a copy.

edit: if this was a registered business with employees in an office setting I would contact police. Even if the police can't help, you need to take care of what you can from a legal standpoint. You don't know how things will play out yet but they may very well end up with you figuring out who did it and seeking damages in court and it's on you to have done what you could.

Not having a backup will work against you in court as it would have minimized damages though... but still. Good luck.

[wheel]

I was owner of a forum that got hacked to the ground. Nothing remained. We started from scratch again immediately, ableit with a few bumps and jars and apologies to the membership. The site continued to flourish, we sold it for six figures and today it's still the largest forum in it's niche.

Sucks to be you big time right now and I expect you've got a difficult few months. But it's NOT a business wipeout, it's a setback. And probably far less of a setback than you think it is.

Get something up there for your visitors to start posting again - they're waiting. Even if it's only a temporary vb install. Figure a permanent fix out later. You've got the traffic and the posters...sucks to lose the old content but the traffic and the posters is the most important thing - that's the core of your business and you didn't lose that.

[wheel]

And my sympathies and understanding over being hacked like this. I appreciate your anger and frustration.

[realmaverick]

Thanks all the great ideas and support. I have asked the host about data recovery. I'm waiting to hear back.

The guy who hacked it, was gloating in an underground hacking forum, asking how much he can sell my member-base for. From that we have his Skype, ICQ and several websites he operates.

He's 19, from the US it appears. My host is also in the US. I'm hoping they'll assist me in taking this as far as I can legally.

We live and learn. I truly believed the backups were safe. The host told me that the server was locked down, inaccessible to the outside world. Then tonight they tell me that isn't the case if a hacker has access to the main servers. Jesus.

The fact my team are pushing me to start again, and the members are emailing with their support, speaks for itself.

I will keep you all updated with the developments and I thank you all once again for such thoughtful, helpful responses.

[martinibuster]

If you don't have a facebook page, start it and keep your members updated. A silver lining in that it will give you the opportunity to grow your social media presence.

Also put a note on your home page about it as well as some vague and general udpates that you are taking action to bring the forum back. Nothing specific, just a communication to keep members advised that you're working on bringing it back. I feel your pain and hope for the best for you.

[slipkid]

No additional technical advice here (other than to second the recommendation to shut down immediately the HD with your data), rather, how I reacted to a FAT failure on one of my company's hard drives. I was determined to recover the data.

I had staff use 4 different data recovery programs to retrieve the data and we got about 80 percent back. Because of a corrupted FAT and that some files had been partially over-written, many recovered files were only half complete and filenames were something like 00001.tmp. It was necessary to go through each file and give it a filename that made sense and delete the mishmash of alphanumerica data that represented the corrupted portion of the file. It took a week to recover what we could.

I was crushed... as if I had lost a close friend. Indeed, the data was correspondence and emails that reflected work product that I was archiving for any future need.

I immediately purchased new computers which became dedicated backups for correspondence and work product.

It took at least a month to get over the fact that I had screwed up. I just could not let go of the disappointment that I had lost what I believed was absolutely vital data. As time went by, I realized that my engineering consulting business was still intact, my clients still had my phone number, and our work product had not suffered at all.

If you are faced with starting from scratch, consider that it is human beings that create value for the business not machines.

[realmaverick]

Good afternoon guys. Thanks so much for all your advice and support.

Today we received confirmation of total data loss. Myself, my team and my members are all devastated. Unfortunately the host didn't have any additional backups. They trawled through all their personal backups the entire night, but due to the size of the website, it was deleted shortly after it was confirmed it wasn't needed.

It's extremely hard to come to terms with. Years, thousands of hours of work, 2 million members, galleries full of members art, their profiles they had so much pride in, all gone.

The host have said, the hacker deleted the data in a way that made data recovery impossible. He targeted our main servers, development servers and backup servers. Apparently a hack to this degree is very rare.

My team are wonderful, passionate and loyal. They want to start again. The members have been sending in tons of support, begging us to rebuild.

Finally, my host has kindly suspended our billing for the foreseeable future, to help us with our recovery. As the servers cost thousands a month, I doubt many hosts would have been so quick to offer their support.

Thanks again to all of you. I will keep you updated with any developments.

[phranque]

you should start collecting urls:

- get a server up now and monitor and analyze all 404s.

- start looking for any cached or scraped pages and see what you can find.

- get some accounts going to use the various inbound link discovery tools <cough>majestic</cough> and add that to the mix.

- provide a forum for your members to supply anything they know of through bookmarks or their own content that links to your site.

[realmaverick]

Great idea Phranque. Thanks!

[aakk9999]

I think you also need to see how the hacker came in. If he was bragging about it, he will see it as a challenge to distroy again anything you are building - often such people will be back as soon as they see you are attempting recovery.

[TheMadScientist]

I'm definitely sorry to hear about this...

I find it hard to believe that whoever did the customization didn't maintain a local copy.

I cannot imagine the person(s) who did the work not having a backup copy of everything, or 10 ... I code ... I usually have one 'complete most recent version' and between 5 and 10 complete copies of previous versions stuffed away somewhere in a zip, never mind all the pieces of incomplete versions I usually have somewhere I could piece together to rebuild an entire project again if I had to.

I would definitely contact the person(s) who did the work and tell them to find a copy of it, because I can about guarantee you: they have one somewhere, even if it's in pieces that need to be reassembled.

But, I would not put it back up until you figure out how the hacker got in, because if it was through the code, without a fix you're making it too easy to do it again.

Also, if the hacker's selling the username list, he has at least that much...
It's yours, Find a Way to Get It.

[TheMadScientist]

The more I think about him having the username list, the more I think I would seriously consider buying it from him...

If you do you not only get your list to get in touch with your members, you usually get a first/last name, address, phone number AND get to set a value on the theft, because it's not 'just hacking' if he downloaded information and resold it.

He stole the username list via the hack...

I would definitely talk to an attorney (or someone who can say definitively) here in the states first, but I'm pretty sure if it's over $5000 it bumps it up from a 'slap on the wrist' to a 'major theft', which means I'd give him $5001 (minimum) for the username list to set a hard, defined value on the theft, then go after him while I was getting things back up and running, but I'm not 'super nice' to people who do things like he did and I think it would be pretty funny to 'play him back a bit', so spending some cash to know he's going to be staring at concrete walls is something I wouldn't mind doing...
.

[edited by: Robert_Charlton at 9:43 pm (utc) on Nov 1, 2012]

[seoskunk]

Nothing is deleted! Its all there still on your server. Ask for the hard drive on the server in the rack and then get a specialist to retrieve your files

[Robert Charlton]

Regarding SEO implications, I'd get a custom 503 unavailable page up there asap, to avoid 404s that might get your backlinks deleted, and to preserve status with the engines. The custom 503 should be a very nice page to explain the situation to all who access your site, provide contact info for your users, etc.

I'm not sure how long you can return a 503, but prolonged 404s are likely to hurt.

[Robert Charlton]

Re other aspects of this...

Nothing is deleted! Its all there still on your server.

Only if not over-written. Not sure whether that has been explored. Yes, it would be wise to consult with a data recovery company. The backup server might be the most recoverable area, but if the hacker was doing it for ransom and he knew what he was doing, it's likely he wiped the data. If you have an auto-backup script, turn it off until you've thoroughly check the backup server. You might have daily and archived backups that weren't touched which can be pieced together.

Try to avoid further activity on your hosting servers that would overwrite what's already there.

Much would depend on how your server was set up or how the disk was partitioned. You'll need to work closely with your web-host, and fast. I'd think that considering the server was hacked, the host would be eager to work with you to avoid further damage.

Definitely explore, as TheMadScientist suggests, getting as much as you can from the people who did the customization. Not sure what content you can get from the WayBack Machine, but it's worth checking.

Cached pages, as phranque suggests, may be a source of page copies. If you were indexed by Blekko, their Premium SEO Tools offer page source and cache. Perhaps they would work with you to make some sort of bulk download available... I don't know. Other "alternative" engines may also have cache data they'd consider making available for bulk download. I'm guessing that the large engines won't, but I don't know that either.

Re the the username list, there's a whole question of what sensitive material the hacker would want to retain make use of even if he did sell it back to you, and how to deal with that.

[phranque]

I'd get a custom 503 unavailable page up there asap

absolutely!
my bad for not thinking of that when i replied previously.
make sure it actually returns a 503 status code.
same as a said above except you will be monitoring 503s instead of 404s.

503 Service Unavailable:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4

[TheMadScientist]

If you don't already have a 503 solution...

### # ###

In the .htaccess file:

RewriteEngine on
RewriteRule .? /serve-503-error.php [L]

OR

In a faster / better place, the httpd.conf file:

RewriteEngine on
RewriteRule . /serve-503-error.php [L]

### # ###

The error header must go at the top of /serve-503-error.php, meaning before any other output (even a space) is sent to the requesting user-agent ... You can run some code before sending the error, but cannot send any output until after the error is served, so the header() lines should go as close to the top of the page as possible.

<serve-503-error.php-below>

<?php

header('HTTP/1.1 503 Service Unavailable');
header('Retry-Afer: '.(86400*DAYS));

### Some more php code to run down here, such as tracking URLs requested and things along those lines.

### Replace DAYS above with the number of days to Retry-After. I would personally go with 7 to 14 in this case, depending on how fast you think you can put things back together a bit. You can always shorten it up later if you get moving along faster.

### There's other ways to code the number to retry-after, but the above is simple to edit.

### The reason I suggest doing it with the internal redirect is you have a bunch of control over things and a ton of flexibility once you get everything running through a separate 503 page, such as, when you add back a few URLs you'll be able to easily check and see if the URL exists and if it does serve the info without the 503 error, otherwise continue serving the 503 error for that URL.

?>

<!-- A Really Cool HTML Page Here for Visitors to See -->

</serve-503-error.php-above>

NOTE: Edited a few times ... Coding off the top of my head without really knowing the entire setup for the site in the future or all the details of the situation I'd want to actually code a solution ... The above should work and be solid, but is it the absolute best in this specific situation? Can't say for sure, but it should be close IMO.

[mark_roach]

Use the strength of your community to help rebuild your site. You might have lost your member list, but your members will still know other members.

You need to get your software back first, someone somewhere must have a copy. Email attachments, usb drives, etc.

Then set the members who have contacted you a challenge "Can we rebuild it bigger and better than before" and emphasise the WE.

Off the top of my head here, but how about making it a competition. Who can contact the most other original members and get them to re-join and upload their stuff.

Thinking about the branches of a tree, could you create something to graphically link and display all the new joiners so they can see the progress that is being made. Person A contacted Person B who contacted Person C, D & E etc.

If the community you have built is half as strong as the one here (and this thread is testament to that) you will have your site back up and running in no time.

[realmaverick]

Thanks again for the amazing support, advice and encouragement.

Just a quickie regarding the 503. Will users be redirected to a single page that returns a 503? Or does it need to be done in a way that the URL remains the same, but a custom 503 page is displayed?

I have read each and every reply and taking it all in to account. Sorry I'm not responding to each, I obviously have a lot to take care of, but I'm listening and appreciating every post. Thanks.