Robots.txt files

[Tom1961]

I notice both my sites have a robots.txt file with this as the only entry:

User-agent: Mediapartners-Google
Disallow:

I don't remember having created this file on either site, and see that Adsense suggests this be removed.

[support.google.com...]


Is the robots.txt file actually hurting my revenue?

[keyplyr]

It is *not* hurting you site nor your Adsense account.

There is no value for Disallow so it is not doing anything.

[fix typo]

[edited by: keyplyr at 8:38 am (utc) on Jun 21, 2018]

[engine]

If that's all it has in the robots.txt file, you don't need it at all.
It shouldn't be hurting your revenue as all it does is specify Google media partners bot is ok to scan your site.
Take it away and it'll do that same.

[phranque]

this tells a Mediapartners-Google user agent it can crawl everything and puts no exclusions on this or any other user agents.

[Tom1961]

Why does adsense say to remove it from the robots.txt file?

[phranque]

the support page you referenced in the OP starts with:

If you’ve modified your site’s robots.txt file to disallow the AdSense crawler from indexing your pages

this is not the case with your robots.txt file, which has been stated 3 different ways above.

[engine]

Why does adsense say to remove it from the robots.txt file?

If that is all your robots.txt file says, just delete the file, you don't need it.

[tangor]

Delete the content of the file, keep the file itself. This avoids having 404s appear in your log files.

As for all the rest, learn more about what robots.txt actually does then make your own decision. As it is it merely says NOTHING IS HAPPENING AND THANKS FOR ALL THE FISH.