AdSense and robots.txt

Is the technical guide incorrect?

[hyperkik]

According to AdSense Support [tinyurl.com]

If your site is using a robots.txt file, the AdSense crawler maybe be blocked from crawling your web pages. Therefore, we may not be able to serve you the most relevant ads based on the content of your website. On pages where we are unable to crawl or understand the content of a page, public service ads may be displayed, for which you will not receive any earnings.

If you would like to grant our crawler access your pages, you can do so without granting permission to any other bots. Simply add the following two lines to the top of your robots.txt file:

User-agent: Mediapartners-Google*
Disallow:

This change will allow our bot to crawl the content of your site and provide you with the most relevant Google ads for your pages.

If you check the effect of your robots.txt file with the Google sitmaps robots.txt tool it appears that you should include:

User-agent: Googlebot-MediaPartners
Disallow:

[leadegroot]

Well when I look at my logs, the user agent I see is

Mediapartners-Google/2.1

Allowing Mediapartners-Google* seems to work for me - it gets into areas other bots are excluded from

[hyperkik]

I don't disagree with following the official advice - personally, I'm including both.