homepage Welcome to WebmasterWorld Guest from 50.19.169.37
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Check my robots.txt please
Don't want to block adsense bot
conroy




msg:1529206
 2:30 pm on Dec 28, 2003 (gmt 0)

Hi all

I am trying to allow google's adsense bot to spider my site so it can target ads to my content. However, I also have a couple pages I do not want indexed. Here is my robots.txt. Could someone please check to make sure I have it right and the adsense bot will spider my site properly and I have blocked the normal googlebot from the page? Thanks!

User-agent: Mediapartners-Google*
Disallow:

User-agent: Googlebot*
Disallow: /resources.html

 

sidyadav




msg:1529207
 3:41 am on Jan 1, 2004 (gmt 0)

conroy, your robots.txt will do the following:
  • allow AdSense bot to all pages
  • dis-allow Googlebot to resources.html
  • allow all other spiders to spider all the pages in your site.

    You can check if your robots.txt file is valid here [searchengineworld.com]

  • butterfingers




    msg:1529208
     2:10 pm on Jan 2, 2004 (gmt 0)

    yes.. that's correct

    Global Options:
     top home search open messages active posts  
     

    Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
    rss feed

    All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
    Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
    WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
    © Webmaster World 1996-2014 all rights reserved