Welcome to WebmasterWorld Guest from 54.163.115.193

Forum Moderators: goodroi

robots.txt-newbie needs help

Is this correct?

   
3:08 am on Nov 16, 2004 (gmt 0)

10+ Year Member



HI All, What I am trying to do here is to allow google's adsense bot,for target ads, and I want to allow all other spiders to spider ALL the pages in my site. Here's my robots.txt:. Is this correct?
Thanking you in advance.

user-agent: mediapartners-Google*
Disallow:

7:52 am on Nov 16, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Yes this is correct it gives mediabot and all other spider access to everything on your site
1:04 pm on Nov 26, 2004 (gmt 0)

10+ Year Member



Wouldn't this:
user-agent: *
Disallow:

accomplish the same thing? You're not actually disallowing any spiders from anywhere in your site, that I can see. Or am I missing something here?

5:18 pm on Nov 26, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



No you are not missing anything, it's what google recommend from within the adsense login to put in your robots.txt file to ensure the pages on your site can be accessed by mediabot.

your code will work the same as will a blank robots.txt file because you are not blocking any spider from accessing your site

 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month