Welcome to WebmasterWorld Guest from 220.127.116.11
After being turned down my site was approved,
Can anybodgy tell me why AdSense frowns on robots.txt?
Please note that you might
not maximize your earnings from AdSense if you have any of the
following on your website:
* forms and dynamic content
* excessive images
* login and password requirements
2. How do I optimize my site for the most relevant ads?
Our ability to target ads to your site depends on the content and structure of your site. Here are some basic guidelines for optimizing your site:
Place ads on pages that predominately contain text -- only text is used to determine a page's context.
If you have a robots.txt file, you'll need to remove it or add the following two lines to your robots.txt to allow our content bot to crawl your site:
might not maximise your earnings
I would take their robots.txt statement only as a recommendation to webmasters as a safe-guard against improper usage of the robots.txt file - to therefore help webmasters earn as much as possible. After reading this thread I did a double check on the sites I have running AdSense and it appears I was lazy on some sites, and diligent on others when it came to including the mediapartners line. AdSense works fine on them all irrespective.
My original take of whothought's post was that he thought AdSense specifically frowned on the usage of a robots.txt file. On the contrary, I think they're stating that line to be helpful, but it's still just a recommendation.
I think they include that there because it is a little extra assurance for some publishers that the mediabot is going to come. And there are plenty of incorrect robots.txt out there which inadvertantly ban all robots - it has been the reason why a couple publishers couldn't get targeted AdSense appearing.
So while it doesn't need to be included, if you do have a robots.txt file, it is always wise to run it through a validator just to be sure your file is doing what you want it to ;)