MediaPartners checking Robots.txt file?

Forum Moderators: martinibuster

Message Too Old, No Replies

MediaPartners checking Robots.txt file?

Has this always been the case?

bhartzer

2:32 pm on Oct 20, 2003 (gmt 0)

I noticed over the weekend that the MediaPartners bot is checking my robots.txt file. Has this always been the case?

jpjones

2:42 pm on Oct 20, 2003 (gmt 0)

Yes, from day 1 AFAIK.

All Google's bots' check for robots.txt.

Normal behaviour is to request robots.txt, then a (or more) web page(s).

bhartzer

2:54 pm on Oct 20, 2003 (gmt 0)

Thanks for making this clear. However, wouldn't it make sense that if I put the Adsense code on a page I'm saying to the bot that it's okay to spider that page and only the pages that have Adsense code on them?