Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Check my robots.txt please

Don't want to block adsense bot

2:30 pm on Dec 28, 2003 (gmt 0)

Junior Member

5+ Year Member

joined:Feb 11, 2008
votes: 0

Hi all

I am trying to allow google's adsense bot to spider my site so it can target ads to my content. However, I also have a couple pages I do not want indexed. Here is my robots.txt. Could someone please check to make sure I have it right and the adsense bot will spider my site properly and I have blocked the normal googlebot from the page? Thanks!

User-agent: Mediapartners-Google*

User-agent: Googlebot*
Disallow: /resources.html

3:41 am on Jan 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 11, 2003
votes: 0

conroy, your robots.txt will do the following:
  • allow AdSense bot to all pages
  • dis-allow Googlebot to resources.html
  • allow all other spiders to spider all the pages in your site.

    You can check if your robots.txt file is valid here [searchengineworld.com]

  • 2:10 pm on Jan 2, 2004 (gmt 0)

    New User

    10+ Year Member

    joined:Jan 2, 2004
    votes: 0

    yes.. that's correct