Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt-newbie needs help

Is this correct?



3:08 am on Nov 16, 2004 (gmt 0)

HI All, What I am trying to do here is to allow google's adsense bot,for target ads, and I want to allow all other spiders to spider ALL the pages in my site. Here's my robots.txt:. Is this correct?
Thanking you in advance.

user-agent: mediapartners-Google*


7:52 am on Nov 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

Yes this is correct it gives mediabot and all other spider access to everything on your site


1:04 pm on Nov 26, 2004 (gmt 0)

10+ Year Member

Wouldn't this:
user-agent: *

accomplish the same thing? You're not actually disallowing any spiders from anywhere in your site, that I can see. Or am I missing something here?


5:18 pm on Nov 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

No you are not missing anything, it's what google recommend from within the adsense login to put in your robots.txt file to ensure the pages on your site can be accessed by mediabot.

your code will work the same as will a blank robots.txt file because you are not blocking any spider from accessing your site


Featured Threads

Hot Threads This Week

Hot Threads This Month