Does robots.txt affect adsense?

psa after psa after psa

[ckanderson]

Does a:

User-agent: *
Disallow: /dev/

have an affect on adsense? I have had few pages that get nothing but PSAs... I'm starting to think that because the google bot hasnt been there that it doesnt know what kind of ads to show?

[Brett_Tabke]

NO. The bot should show up after you have ads on the page.

Robots.txt has little to do with adsense (as long as the spider isn't blocked from the pages you are putting adsense on).

[ckanderson]

Ok, thats good to know... I'll just have to wait for the bot to come by and see what my site is all about.

As for blocking the bot...

User-agent: *
Disallow: /dev/

As far as I know that sets it so that whatever is in my /dev/ directory isn't indexed by the bots... does that also mean that they are totally being blocked?

[jdMorgan]

Blocked? No, robots.txt doesn't block anything. It is a polite request to presumably-polite spiders to please stay out of the specified paths -- In your case /dev/.

Unless your pages that are showing PSAs are in the /dev/ path, this will have no effect on Google's Mediapartners-Google 'bot.

Jim

[Jenstar]

You won't show any ads in the /dev/ directory, because the AdSense bot (that Google uses to determine the content on your page to serve ads) does obey robots.txt. So if you are banning all bots from certain pages or directories, AdSense will only show PSAs on those pages/directories.

If you are having problems on pages that don't reside in the /dev/ directory, it could be a stop word issue, if your pages have been online for more than a few hours. You can also check your logs for the media partners bot to make sure it is stopping by to grab the content on those pages.

[buckworks]

I'd urge you to make use of the alternate ads feature, so that when regular Adsense ads aren't forthcoming the alternate could be something of your choice instead of a PSA.

[ckanderson]

The pages that I'm showing the ads on are in the /dev/ directory, so the bot obeying robots.txt explains it now.

I'll just have to put up with the psa's until I move everything to the main directory.

Thanks!

[Jenstar]

There is more about allowing the AdSense bot using robots.txt here:
[google.com...]

[arrowman]

I'll just have to put up with the psa's until I move everything to the main directory.

Alternatively, you can grant this particular bot access to /dev/. See the URL in Jenstar's posting.

[ckanderson]

Implemented that and now it works perfect!

Thanks!