Welcome to WebmasterWorld Guest from 107.20.34.173

Forum Moderators: goodroi

Message Too Old, No Replies

Disallow all except adsense

     

thegreatpretender

1:58 pm on Oct 20, 2007 (gmt 0)

10+ Year Member



Hi,
I would like to disallow all bots except adsense, this is what my robots.txt looks like:

User-agent: *
Disallow: /

User-agent: Googlebot-Mobile
Disallow: /

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Is this correct?

Thanks

goodroi

1:41 pm on Oct 23, 2007 (gmt 0)

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



That is correct according to Google's robot.txt checker. If you ever want to test how Google will respond to your robots.txt you can use their tools in their Webmaster Central area.

jimbeetle

3:06 pm on Oct 23, 2007 (gmt 0)

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member



You could simplify it:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /

Note that it's just my preference to put 'bot-specific directives before catchalls, and I don't use the Allow as it's non-standard. In this case, of course, the Mediapartners bot would obey it; I just don't want to keep track of which 'bots obey it or not.

thegreatpretender

5:47 pm on Oct 23, 2007 (gmt 0)

10+ Year Member



Thanks very much.

ecmedia

1:53 pm on Nov 15, 2007 (gmt 0)

5+ Year Member



I have the same issue and I just followed the guidelines above. How long does it take to show results? I just launched a website and initially I just blocked Googlebot but then I did not see any AdSense ads. Now I have allowed MediaPartners-Google and when can I expect to see ads?

ecmedia

4:40 pm on Nov 19, 2007 (gmt 0)

5+ Year Member



Update: AdSense bot came almost instantly and I see ads just fine.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month