homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google AdSense
Forum Library, Charter, Moderators: incrediBILL & jatar k & martinibuster

Google AdSense Forum

AdSense robots.txt
why AdSense frowns on robots.txt?

 7:39 pm on Dec 29, 2003 (gmt 0)

Hello World

After being turned down my site was approved,

Can anybodgy tell me why AdSense frowns on robots.txt?

Please note that you might
not maximize your earnings from AdSense if you have any of the
following on your website:
* robots.txt
* frames
* forms and dynamic content
* excessive images
* login and password requirements




 7:43 pm on Dec 29, 2003 (gmt 0)

I'd guess that the concern is that you might have an improperly configured robots.txt, which is not uncommon. So it might, for example, keep the mediapartners bot from crawling your site.

As long as your robots.txt is in order it shouldn't have any negative impact on AdSense.


 11:12 pm on Dec 29, 2003 (gmt 0)

Your robots.txt file must contain this

User-agent: Mediapartners-Google*

hope this helps



 11:32 pm on Dec 29, 2003 (gmt 0)

Your robots.txt file must contain this

"must"? My robots.txt files have no robots being excluded (or specifically included) and I have had no issues with AdSense appearing on my pages. whothought might have excluded all robots in his original version.


 11:43 pm on Dec 29, 2003 (gmt 0)

That is what google state on the adsense FAQ page

program basic's
2. How do I optimize my site for the most relevant ads?
Our ability to target ads to your site depends on the content and structure of your site. Here are some basic guidelines for optimizing your site:

Place ads on pages that predominately contain text -- only text is used to determine a page's context.
If you have a robots.txt file, you'll need to remove it or add the following two lines to your robots.txt to allow our content bot to crawl your site:

User-agent: Mediapartners-Google*



 12:00 am on Dec 30, 2003 (gmt 0)

might not maximise your earnings

I would take their robots.txt statement only as a recommendation to webmasters as a safe-guard against improper usage of the robots.txt file - to therefore help webmasters earn as much as possible. After reading this thread I did a double check on the sites I have running AdSense and it appears I was lazy on some sites, and diligent on others when it came to including the mediapartners line. AdSense works fine on them all irrespective.

My original take of whothought's post was that he thought AdSense specifically frowned on the usage of a robots.txt file. On the contrary, I think they're stating that line to be helpful, but it's still just a recommendation.



 12:04 am on Dec 30, 2003 (gmt 0)

I can't believe this thread exists but I'm not judging anyone.

It is merely a guideline, folks. If you use a robots.txt validator you should be good to go.


 8:38 pm on Dec 30, 2003 (gmt 0)

thanx for the help.
if had read the faq page @ adsense i would have found the answer.

Oaf357 sorry i thought it was a fair question.



 8:46 pm on Dec 30, 2003 (gmt 0)

It was a fair question
if you don't know the answer ask, there are lots of people here who are willing to help

By the way whothought I forgot my manners
Welcome to Webmaster World



 8:47 pm on Dec 30, 2003 (gmt 0)

I don't specifically mention the mediabot in some of my sites, and it hasn't affected me adversely.

I think they include that there because it is a little extra assurance for some publishers that the mediabot is going to come. And there are plenty of incorrect robots.txt out there which inadvertantly ban all robots - it has been the reason why a couple publishers couldn't get targeted AdSense appearing.

So while it doesn't need to be included, if you do have a robots.txt file, it is always wise to run it through a validator just to be sure your file is doing what you want it to ;)


 8:50 pm on Dec 30, 2003 (gmt 0)

Your right jenstar I was stating what was on the google adsense FAQ's for any newbie's who may not understand the full ins and outs of a robots.txt file and through no fault of their own ban mediabot from gaining access to their sites


Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google AdSense
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved