Welcome to WebmasterWorld Guest from 54.147.44.93

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

AdSense robots.txt

why AdSense frowns on robots.txt?

     

whothought

7:39 pm on Dec 29, 2003 (gmt 0)

10+ Year Member



Hello World

After being turned down my site was approved,

Can anybodgy tell me why AdSense frowns on robots.txt?

Please note that you might
not maximize your earnings from AdSense if you have any of the
following on your website:
* robots.txt
* frames
* forms and dynamic content
* excessive images
* login and password requirements

Michael

JayC

7:43 pm on Dec 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I'd guess that the concern is that you might have an improperly configured robots.txt, which is not uncommon. So it might, for example, keep the mediapartners bot from crawling your site.

As long as your robots.txt is in order it shouldn't have any negative impact on AdSense.

ncw164x

11:12 pm on Dec 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Your robots.txt file must contain this

User-agent: Mediapartners-Google*
Disallow:

hope this helps

ncw164x

anallawalla

11:32 pm on Dec 29, 2003 (gmt 0)

WebmasterWorld Administrator anallawalla is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Your robots.txt file must contain this

"must"? My robots.txt files have no robots being excluded (or specifically included) and I have had no issues with AdSense appearing on my pages. whothought might have excluded all robots in his original version.

ncw164x

11:43 pm on Dec 29, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



That is what google state on the adsense FAQ page

program basic's
2. How do I optimize my site for the most relevant ads?
Our ability to target ads to your site depends on the content and structure of your site. Here are some basic guidelines for optimizing your site:

Place ads on pages that predominately contain text -- only text is used to determine a page's context.
If you have a robots.txt file, you'll need to remove it or add the following two lines to your robots.txt to allow our content bot to crawl your site:

User-agent: Mediapartners-Google*
Disallow

ncw164x

2oddSox

12:00 am on Dec 30, 2003 (gmt 0)

10+ Year Member



might not maximise your earnings

I would take their robots.txt statement only as a recommendation to webmasters as a safe-guard against improper usage of the robots.txt file - to therefore help webmasters earn as much as possible. After reading this thread I did a double check on the sites I have running AdSense and it appears I was lazy on some sites, and diligent on others when it came to including the mediapartners line. AdSense works fine on them all irrespective.

My original take of whothought's post was that he thought AdSense specifically frowned on the usage of a robots.txt file. On the contrary, I think they're stating that line to be helpful, but it's still just a recommendation.

2odd...

Oaf357

12:04 am on Dec 30, 2003 (gmt 0)

10+ Year Member



I can't believe this thread exists but I'm not judging anyone.

It is merely a guideline, folks. If you use a robots.txt validator you should be good to go.

whothought

8:38 pm on Dec 30, 2003 (gmt 0)

10+ Year Member



thanx for the help.
if had read the faq page @ adsense i would have found the answer.

Oaf357 sorry i thought it was a fair question.

mike

ncw164x

8:46 pm on Dec 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



It was a fair question
if you don't know the answer ask, there are lots of people here who are willing to help

By the way whothought I forgot my manners
Welcome to Webmaster World

ncw164x

Jenstar

8:47 pm on Dec 30, 2003 (gmt 0)

WebmasterWorld Senior Member jenstar is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I don't specifically mention the mediabot in some of my sites, and it hasn't affected me adversely.

I think they include that there because it is a little extra assurance for some publishers that the mediabot is going to come. And there are plenty of incorrect robots.txt out there which inadvertantly ban all robots - it has been the reason why a couple publishers couldn't get targeted AdSense appearing.

So while it doesn't need to be included, if you do have a robots.txt file, it is always wise to run it through a validator just to be sure your file is doing what you want it to ;)

ncw164x

8:50 pm on Dec 30, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Your right jenstar I was stating what was on the google adsense FAQ's for any newbie's who may not understand the full ins and outs of a robots.txt file and through no fault of their own ban mediabot from gaining access to their sites

ncw164x

 

Featured Threads

Hot Threads This Week

Hot Threads This Month