homepage Welcome to WebmasterWorld Guest from 54.163.139.36
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Disallow all except adsense
thegreatpretender

10+ Year Member



 
Msg#: 3482706 posted 1:58 pm on Oct 20, 2007 (gmt 0)

Hi,
I would like to disallow all bots except adsense, this is what my robots.txt looks like:

User-agent: *
Disallow: /

User-agent: Googlebot-Mobile
Disallow: /

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Is this correct?

Thanks

 

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3482706 posted 1:41 pm on Oct 23, 2007 (gmt 0)

That is correct according to Google's robot.txt checker. If you ever want to test how Google will respond to your robots.txt you can use their tools in their Webmaster Central area.

jimbeetle

WebmasterWorld Senior Member jimbeetle us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3482706 posted 3:06 pm on Oct 23, 2007 (gmt 0)

You could simplify it:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /

Note that it's just my preference to put 'bot-specific directives before catchalls, and I don't use the Allow as it's non-standard. In this case, of course, the Mediapartners bot would obey it; I just don't want to keep track of which 'bots obey it or not.

thegreatpretender

10+ Year Member



 
Msg#: 3482706 posted 5:47 pm on Oct 23, 2007 (gmt 0)

Thanks very much.

ecmedia

5+ Year Member



 
Msg#: 3482706 posted 1:53 pm on Nov 15, 2007 (gmt 0)

I have the same issue and I just followed the guidelines above. How long does it take to show results? I just launched a website and initially I just blocked Googlebot but then I did not see any AdSense ads. Now I have allowed MediaPartners-Google and when can I expect to see ads?

ecmedia

5+ Year Member



 
Msg#: 3482706 posted 4:40 pm on Nov 19, 2007 (gmt 0)

Update: AdSense bot came almost instantly and I see ads just fine.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved