HI All, What I am trying to do here is to allow google's adsense bot,for target ads, and I want to allow all other spiders to spider ALL the pages in my site. Here's my robots.txt:. Is this correct? Thanking you in advance.
user-agent: mediapartners-Google* Disallow:
7:52 am on Nov 16, 2004 (gmt 0)
Yes this is correct it gives mediabot and all other spider access to everything on your site
1:04 pm on Nov 26, 2004 (gmt 0)
Wouldn't this: user-agent: * Disallow:
accomplish the same thing? You're not actually disallowing any spiders from anywhere in your site, that I can see. Or am I missing something here?
5:18 pm on Nov 26, 2004 (gmt 0)
No you are not missing anything, it's what google recommend from within the adsense login to put in your robots.txt file to ensure the pages on your site can be accessed by mediabot.
your code will work the same as will a blank robots.txt file because you are not blocking any spider from accessing your site