Welcome to WebmasterWorld Guest from 188.8.131.52 , register , login , search , subscribe , help , library , PubCon , announcements , recent posts , open posts Subscribe and Support WebmasterWorld
robots.txt-newbie needs help Is this correct? cybervette msg:1526084 3:08 am on Nov 16, 2004 (gmt 0) HI All, What I am trying to do here is to allow google's adsense bot,for target ads, and I want to allow all other spiders to spider ALL the pages in my site. Here's my robots.txt:. Is this correct? Thanking you in advance.
ncw164x msg:1526085 7:52 am on Nov 16, 2004 (gmt 0)
Yes this is correct it gives mediabot and all other spider access to everything on your site sonjay msg:1526086 1:04 pm on Nov 26, 2004 (gmt 0)
Wouldn't this: user-agent: * Disallow:
accomplish the same thing? You're not actually disallowing any spiders from anywhere in your site, that I can see. Or am I missing something here?
ncw164x msg:1526087 5:18 pm on Nov 26, 2004 (gmt 0)
No you are not missing anything, it's what google recommend from within the adsense login to put in your robots.txt file to ensure the pages on your site can be accessed by mediabot.
your code will work the same as will a blank robots.txt file because you are not blocking any spider from accessing your site