Website hacked - robots blocked googlebot - has this affected ads?

hacked website, google adsense

[zarathustra2011]

When I heard about lots of people having problems with blank ads on the 20th January, I assumed my problem was the same, but now it seems that was just a red herring. My website was hacked on the 20th January, and the devils who did it added new directories such as kids-boots, ugg-boots, along with ASP pages and code.
They hadn't changed my website itself, but I noticed my robots file had been changed, and all search engine bots were blocked. I've since changed it, along with all my passwords.
If the robots file was manipulated on the 20th January, might it have taken 3 days to take affect? I noticed a 50% drop in traffic on the 23rd January. Would the block also mean google is unable to target my website, hence why many of my pages were unable to feature any adverts? For the pages that still worked, would that be because those pages were still in google's cache? Also, does anyone know how often Google typical checks a robots file, so I have a rough idea when things may hopefully go back to normal?

[Marketing Guy]

Same thing happened to my site and also coincided with my (now former) hosting company going silent on me. Hacked site was live for several weeks with each page hosting to hidden links to Ugg boots.

Adsense / robots.txt weren't affected (although the amount of time I spent on the site fixing it and transfer it to a new CMS meant Google started retargeting Ugg boots ads to me on other sites...). Traffic was fine - no negative impact that I could see.

Google grabs robots.txt regularly - login to Google webmaster tools to see the last time Google downloaded yours. You could also submit the robots.txt via the "fetch as Googlebot" option to get a fresh copy updated.

Things should sort themselves out. Timeframe depends on how frequently Google indexes your site, but you could submit your XML sitemap and ramp up the crawl speed / use the fetch tool to speed things along nicely.

[zarathustra2011]

Sorry to hear it Marketing Guy - normally it wouldn't have affected my traffic, but because they blocked me from the search engines by overwriting my robots.txt, I've fortunately discovered it relatively early on.

I've clicked on fetch as googlebot, but under fetch status it says 'Denied by robots.txt'. If I click on the file I fetched, it displays the old reuploaded robots.txt that I put up 3 hours ago. I did ramp up the crawl speed, but I think I might put it on maximum. Just hope I don't lose my number one status in Google under certain keywords.

[Marketing Guy]

Your robots.txt file blocks itself? I just tried submitting my own and it was fetched fine. Do you use any wildcards that might be accidentally blocking the robots.txt?

You shouldn't lose any rankings over a technical glitch. I went backpacking around South America, maxed one of my cards and my hosting company shut down one of my sites because I forgot to pay them (oops). Was down for over a week. It killed the Adsense revenue as the placement advertisers ran off, but rankings recovered when the site went back online.

[Marketing Guy]

Put your site's robots.txt URL into the crawler access tool and see which line is preventing Google from accessing it.

[zarathustra2011]

Thanks again Marketing Guy - you're reassuring me here.

Crawler access is still showing the hackers own robot file. I can copy and paste mine, but it's only for testing purposes.

Part of their robot file reads:
User-agent: *
Allow: /ugg-boots/
Allow: /kids-boots/
Allow: /women-boots/
Allow: /mens-boots/
Allow: /boots/
Disallow: /

I guess I just have to wait until Google grabs my updated robots.txt -last time it got it was 6 hours ago. I updated the robots.txt about 3.5 hours ago.

[zarathustra2011]

Sorry, I had meant to say, that on the face of it, it does seem the robots file is blocking itself! Don't know how to get around that until Google grabs my reuploaded robots.txt

Under the general webmaster tools log in, it does state - Severe health issues are found on your site. It tells me robots.txt has blocked my site, and then it gives a link to my website's main homepage.

[Marketing Guy]

That message is a legacy from the old robots.txt file so don't worry about that. Had the same thing last week with a subdirectory I removed.

Give it 24 hours - Google will catch up eventually. :)

[zarathustra2011]

I hope so Marketing Guy. As I can't fetch anything, I began to panic and thought maybe the bots will not even bother trying to read my reuploaded robots.txt as my site has been blocked. I can't resubmit my feed, as that is also blocked by robots.txt it seems.

I'm sure I'm panicking for nothing - just very eager for it to return as normal. It's been a costly experience.

[Marketing Guy]

The fetch tool for some reason uses the previously downloaded robots.txt and not the live one. All will be fine. :)

[zarathustra2011]

Thanks again - your replies really help take the stress out of things. I'll sit tight. Never been hacked before, so I'm pretty peeved it happened.

[Marketing Guy]

No problem at all - it can be a headache. It's just the not knowing what'll happen more than anything. I've had someone post 10k+ #*$!o images on one of my discussion forums on Xmas day once. That was fun. :) These things sort themselves out easily enough.

[zarathustra2011]

I suppose they could have done more damage - at least they left the site itself intact; just wish they hadn't messed with my source of revenue. The worst I had was a year long campaign a couple of years back, where Nigerian scammers were sending out emails (and posting on job forums) listing bogus jobs, and mentioning my website in their mass mailing with a false London address. Had to write a lot of replies telling people it had nothing to do with my website. Now if I search for my site, some of the major fraud warning sites pop up high in the list, which is not a nice association to have, even though I've responded on the sites to clear things up. That's the sort of scam, you can't protect yourself against.

What an awful Christmas you must have had! There's some nasty individuals out there.

[HuskyPup]

@zarathustra2011

And what does your web host have to say about this?

[netmeg]

First of all - plug the hole.

Second of all - check your log files, and see who has been accessing (or trying to access) your robots.txt file in the past 24 hours.

Third of all - do a site:yourdomain.com in Google to see what shows up.

[zarathustra2011]

Husky - I went on their livehelp and they told me to send it as a ticket, so I'm still waiting to hear back from them. Normally they get back to me within the hour, but I've heard nothing from them this afternoon.

Netmeg, I've been trying to download my log for that day, but I don't know if it's my FTP software - at the moment it just doesn't want to download.

[zarathustra2011]

There's a tiny log file on the 21st January that looks suspicious (normally my log files are very big), amongst the other logs. There's an IP address in there that comes from China, and I suspect that's the culprit.
Without posting it all, there's some wierd stuff in there I don't understand.


GET /goose.asp key=real%20goose%20jackets&key2=2080&key3=1192 200 0 24453 277 297 HTTP/1.1 Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)

[netmeg]

Well Baidu is a large (and well known) search engine in Asia.

[Play_Bach]

> and the devils who did it added new directories such as kids-boots, ugg-boots, along with ASP pages and code.

Consider yourself lucky. About ten years ago, I found naked people on my site thanks to some JavaScript (originating from Russia) that had made it's way onto my hosting company's servers.

[zarathustra2011]

Well, I guess it's a Chinese hacker - not that I can do much with that information. At least it's a chance to make my password more secure.

Play_Bach - tasteful nudes I hope!

[Play_Bach]

> Play_Bach - tasteful nudes I hope!

uh... no. :-|

[zarathustra2011]

Just hope you didn't have any children, niece or nephews taking a look at your website on that day!

[Play_Bach]

> Just hope you didn't have any children, niece or nephews taking a look at your website on that day!

Don't think so, but if they did, they sure got an education! In my case, it was many, many sites that got affected because the crooks had broken into the back end of the hosting company itself, so the problem was server wide. Nothing any of us could do about it really, except wait for the hosting company to figure out a fix.

[zarathustra2011]

Must have been a fun time for that hosting company. :-)

[zarathustra2011]

My Robots.txt has been restored and I've been able to resubmit my sitemap. Just hope my website fills up with relevant ads quickly and I get my lost traffic back.

[zarathustra2011]

I have a query I hope somebody can help me with. I now have ads coming back to my website, but my traffic at the moment is still extremely low.
Under Google's webmaster tools and 'keywords' it says the most popular keywords are goose, canada, parka, jackets. My site is art related, and these keywords only appeared when the hacker inserted their junk on my website. I found this in my log files:

GET /goose.asp key=real%20goose%20jackets&key2=2080&key3=1192 200 0 24453 277 297 HTTP/1.1 Mozilla/5.0+(compatible;+Baiduspider/2.0;++http://www.baidu.com/search/spider.html)


Now it's gone, do I just have to wait for Google to eventually reindex my pages and update itself with my actual keywords, as I fear it's harmed my traffic (which has been reduced by 75% now). I've set my crawl rate to maximum for the moment, but am not sure if there's anything else I can do other than to wait?

[zarathustra2011]

Also just to add, if I search for the keywords my site used to reach number one position for in Google. I'm still there, but lower down on the first page, however the title and description google are using are from a very old version of the site. I can only assume they're using a very old cached version?!

[eddieh]

Try pinging your site through Pingomatic and/or Feedshark. Doing so will at least alert the bots that your site has been updated.

Even if you are using Wordpress, I have found that separate pinging (as opposed to relying on the internal wordpress pinging) tends to be more effective at alerting the bots

[zarathustra2011]

Thanks Eddieh. I'm trying everything at the moment, but I think I just have to sit tight and wait. The reuploaded robots got updated yesterday, but I have an 80% decrease in traffic today and Adsense earnings are absolutely pitiful. Just got fingers and toes crossed that everything goes back to normal in a day or two, though I appreciate it may take longer.

[Play_Bach]

> though I appreciate it may take longer.

It's been my experience that the search engines are slow to update, sometimes taking months to restore things. Hopefully, that won't be the case here.