Welcome to WebmasterWorld Guest from 54.167.102.69

Forum Moderators: goodroi

Message Too Old, No Replies

Disallow all except adsense

     
1:58 pm on Oct 20, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 12, 2004
posts:550
votes: 0


Hi,
I would like to disallow all bots except adsense, this is what my robots.txt looks like:

User-agent: *
Disallow: /

User-agent: Googlebot-Mobile
Disallow: /

User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: Mediapartners-Google
Allow: /

Is this correct?

Thanks

1:41 pm on Oct 23, 2007 (gmt 0)

Administrator from US 

WebmasterWorld Administrator goodroi is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:June 21, 2004
posts:3080
votes: 67


That is correct according to Google's robot.txt checker. If you ever want to test how Google will respond to your robots.txt you can use their tools in their Webmaster Central area.
3:06 pm on Oct 23, 2007 (gmt 0)

Senior Member

WebmasterWorld Senior Member jimbeetle is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Oct 26, 2002
posts:3292
votes: 6


You could simplify it:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /

Note that it's just my preference to put 'bot-specific directives before catchalls, and I don't use the Allow as it's non-standard. In this case, of course, the Mediapartners bot would obey it; I just don't want to keep track of which 'bots obey it or not.

5:47 pm on Oct 23, 2007 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 12, 2004
posts:550
votes: 0


Thanks very much.
1:53 pm on Nov 15, 2007 (gmt 0)

Preferred Member

5+ Year Member

joined:Sept 28, 2007
posts:487
votes: 0


I have the same issue and I just followed the guidelines above. How long does it take to show results? I just launched a website and initially I just blocked Googlebot but then I did not see any AdSense ads. Now I have allowed MediaPartners-Google and when can I expect to see ads?
4:40 pm on Nov 19, 2007 (gmt 0)

Preferred Member

5+ Year Member

joined:Sept 28, 2007
posts:487
votes: 0


Update: AdSense bot came almost instantly and I see ads just fine.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members