AdSense robots.txt

why AdSense frowns on robots.txt?

[whothought]

Hello World

After being turned down my site was approved,

Can anybodgy tell me why AdSense frowns on robots.txt?

Please note that you might
not maximize your earnings from AdSense if you have any of the
following on your website:
* robots.txt
* frames
* forms and dynamic content
* excessive images
* login and password requirements

Michael

[JayC]

I'd guess that the concern is that you might have an improperly configured robots.txt, which is not uncommon. So it might, for example, keep the mediapartners bot from crawling your site.

As long as your robots.txt is in order it shouldn't have any negative impact on AdSense.

[ncw164x]

Your robots.txt file must contain this

User-agent: Mediapartners-Google*
Disallow:

hope this helps

ncw164x

[anallawalla]

Your robots.txt file must contain this

"must"? My robots.txt files have no robots being excluded (or specifically included) and I have had no issues with AdSense appearing on my pages. whothought might have excluded all robots in his original version.

[ncw164x]

That is what google state on the adsense FAQ page

program basic's
2. How do I optimize my site for the most relevant ads?
Our ability to target ads to your site depends on the content and structure of your site. Here are some basic guidelines for optimizing your site:

Place ads on pages that predominately contain text -- only text is used to determine a page's context.
If you have a robots.txt file, you'll need to remove it or add the following two lines to your robots.txt to allow our content bot to crawl your site:

User-agent: Mediapartners-Google*
Disallow

ncw164x

[2oddSox]

might not maximise your earnings

I would take their robots.txt statement only as a recommendation to webmasters as a safe-guard against improper usage of the robots.txt file - to therefore help webmasters earn as much as possible. After reading this thread I did a double check on the sites I have running AdSense and it appears I was lazy on some sites, and diligent on others when it came to including the mediapartners line. AdSense works fine on them all irrespective.

My original take of whothought's post was that he thought AdSense specifically frowned on the usage of a robots.txt file. On the contrary, I think they're stating that line to be helpful, but it's still just a recommendation.

2odd...

[Oaf357]

I can't believe this thread exists but I'm not judging anyone.

It is merely a guideline, folks. If you use a robots.txt validator you should be good to go.

[whothought]

thanx for the help.
if had read the faq page @ adsense i would have found the answer.

Oaf357 sorry i thought it was a fair question.

mike

[ncw164x]

It was a fair question
if you don't know the answer ask, there are lots of people here who are willing to help

By the way whothought I forgot my manners
Welcome to Webmaster World

ncw164x

[Jenstar]

I don't specifically mention the mediabot in some of my sites, and it hasn't affected me adversely.

I think they include that there because it is a little extra assurance for some publishers that the mediabot is going to come. And there are plenty of incorrect robots.txt out there which inadvertantly ban all robots - it has been the reason why a couple publishers couldn't get targeted AdSense appearing.

So while it doesn't need to be included, if you do have a robots.txt file, it is always wise to run it through a validator just to be sure your file is doing what you want it to ;)

[ncw164x]

Your right jenstar I was stating what was on the google adsense FAQ's for any newbie's who may not understand the full ins and outs of a robots.txt file and through no fault of their own ban mediabot from gaining access to their sites

ncw164x