Delisted from Google because of AdSense Bot

Mediapartners-Google/2.1 (+http://www.googlebot.com/bot.html)

[Lisa]

Media/GoogleBot has kicked our main website out of the index (note: deep content is still listed, but "/" is totally gone). The homepage can no longer be found, I believe this all stems from the launch of AdSense on our site at about 1PM today, We received 5000 ad impressions the first 3 hours. So we were rather happy. Then came this Mediapartners robot from Google to index all the content that was previously hidden to Google but only visible to humans. The regular GoogleBot is forbidden to look at this content and doesn't ask for it. But this new robot thinks it is better then the robots.txt directives (even though it asked for the robots.txt a few times). So off this robot went querying away at forbidden content. But as with all robots in the human only content we redirect them back to the front page. So we also directed this Media robot as well. But now, I think after several thousand lookups to forbidden content and getting redirected back to "/" Google has yanked our site from the main Google index.

So I think this is a real bug that needs to be fixed. Just a word of advice, the adsense thing is cool. But this little incident is rather disturbing and I hope it gets solved soon.

I am debaiting giving this robot access as content has it is already cached, but I don't want this content appearing in the Google Index next month or as part of FreshDeepMediaBot update. So, does AdSense cause changes in the regular index? YES. I have only see negitive results myself, but I am sure that AdSense is another way to call FreshDeepMediaBot.

:(

[ciml]

Lisa, did you have a line in your REP file for the AdSense, or did you forbid 'all'? If neither, maybe it responds to its own name instead.

I normally look towards the possibility of coincidence in these types of cases, but if the Mediapartners robots is integrated with Googlebot then we should all think about the implications.

[Clark]

This definitely brings out that tingling conspiracy theory sense in all of us. Makes a lot of sense. Thank you for sharing and I'll be watching intently on updates to this matter.

[GoogleGuy]

Urg. I wish every third message didn't mention conspiracy theories around here lately. ;) Lisa, we gave the bots different names so that you could use robots.txt on one but not the other. The AdSense bots and Googlebot have never met, and don't exchange information, so they're completely independent. I'll ask around about this, but to show content ads we normally have to have or fetch the content of the page.

If you've got 30M pages, none of which Google is allowed to visit, then AdSense might not be the best match for your sites for now. You might want to remove the AdSense JavaScript for the time being while we communicate about this.

[Clark]

Sorry GG. I knew you'd come in and clear it up one way or another. I just felt bad for Lisa because it's easy for any of us to see ourselves in her situation. And quite a nightmare to imagine joining adsense would have a *negative* effect.

For me the whole conspiracy thing is an amusing joke. I got quite a kick out of that guy's website because I thought he did a very crafty job of making his case. I don't really believe it, I've seen Brett make some pretty strong statements that just don't jive with his case, but it does display some writing skill...

P.S. Just noticed that Lisa is a moderator here!

[ciml]

Thanks for clearing that up GoogleGuy, so it looks like coincidence then.

[Lisa]

Well, I guess this would be partly our fault for redirecting googlebot away from the content several thousand times over and over again.... So I think we will modify the pages to give google some of the non-network information. But I know for sure the thousands of redirection of the media robot do effect the regular index. I just didn't think there was a connection.

[polarmate]

You want to display content-driven ads on a page whose content you don't want indexed? Kind of contradictory, isn't it? Even if the bots are not the same...

[Lisa]

OK, MediaBot now receives the same content the last real user saw, so there is no need to redirect MediaBot, but my main concern is that MediaBot is MediaDeepFreshCrawlBot. Will this content make its way into the regular index? And how long does a MediaBot penality take you out of the index. Would Fresh bot re-add us, or is there a permanite penality that MediaBot gave us?