homepage Welcome to WebmasterWorld Guest from 54.242.18.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Disallow all except adsense
thegreatpretender




msg:3482708
 1:58 pm on Oct 20, 2007 (gmt 0)

Hi,
I would like to disallow all bots except adsense, this is what my robots.txt looks like:

User-agent: *
Disallow: /

User-agent: Googlebot-Mobile
Disallow: /

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Is this correct?

Thanks

 

goodroi




msg:3484989
 1:41 pm on Oct 23, 2007 (gmt 0)

That is correct according to Google's robot.txt checker. If you ever want to test how Google will respond to your robots.txt you can use their tools in their Webmaster Central area.

jimbeetle




msg:3485064
 3:06 pm on Oct 23, 2007 (gmt 0)

You could simplify it:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /

Note that it's just my preference to put 'bot-specific directives before catchalls, and I don't use the Allow as it's non-standard. In this case, of course, the Mediapartners bot would obey it; I just don't want to keep track of which 'bots obey it or not.

thegreatpretender




msg:3485262
 5:47 pm on Oct 23, 2007 (gmt 0)

Thanks very much.

ecmedia




msg:3505664
 1:53 pm on Nov 15, 2007 (gmt 0)

I have the same issue and I just followed the guidelines above. How long does it take to show results? I just launched a website and initially I just blocked Googlebot but then I did not see any AdSense ads. Now I have allowed MediaPartners-Google and when can I expect to see ads?

ecmedia




msg:3508921
 4:40 pm on Nov 19, 2007 (gmt 0)

Update: AdSense bot came almost instantly and I see ads just fine.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved