homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Disallow all except adsense

 1:58 pm on Oct 20, 2007 (gmt 0)

I would like to disallow all bots except adsense, this is what my robots.txt looks like:

User-agent: *
Disallow: /

User-agent: Googlebot-Mobile
Disallow: /

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Is this correct?




 1:41 pm on Oct 23, 2007 (gmt 0)

That is correct according to Google's robot.txt checker. If you ever want to test how Google will respond to your robots.txt you can use their tools in their Webmaster Central area.


 3:06 pm on Oct 23, 2007 (gmt 0)

You could simplify it:

User-agent: Mediapartners-Google

User-agent: *
Disallow: /

Note that it's just my preference to put 'bot-specific directives before catchalls, and I don't use the Allow as it's non-standard. In this case, of course, the Mediapartners bot would obey it; I just don't want to keep track of which 'bots obey it or not.


 5:47 pm on Oct 23, 2007 (gmt 0)

Thanks very much.


 1:53 pm on Nov 15, 2007 (gmt 0)

I have the same issue and I just followed the guidelines above. How long does it take to show results? I just launched a website and initially I just blocked Googlebot but then I did not see any AdSense ads. Now I have allowed MediaPartners-Google and when can I expect to see ads?


 4:40 pm on Nov 19, 2007 (gmt 0)

Update: AdSense bot came almost instantly and I see ads just fine.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved