Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Check my robots.txt please Don't want to block adsense bot conroy msg:1529206 2:30 pm on Dec 28, 2003 (gmt 0) Hi all
I am trying to allow google's adsense bot to spider my site so it can target ads to my content. However, I also have a couple pages I do not want indexed. Here is my robots.txt. Could someone please check to make sure I have it right and the adsense bot will spider my site properly and I have blocked the normal googlebot from the page? Thanks!
sidyadav msg:1529207 3:41 am on Jan 1, 2004 (gmt 0)
conroy, your robots.txt will do the following: allow AdSense bot to all pages dis-allow Googlebot to resources.html allow all other spiders to spider all the pages in your site.
You can check if your robots.txt file is valid
here [ searchengineworld.com] butterfingers msg:1529208 2:10 pm on Jan 2, 2004 (gmt 0)
yes.. that's correct