Welcome to WebmasterWorld Guest from 23.22.220.37

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt-newbie needs help

Is this correct?

     

cybervette

3:08 am on Nov 16, 2004 (gmt 0)

Inactive Member
Account Expired

 
 


HI All, What I am trying to do here is to allow google's adsense bot,for target ads, and I want to allow all other spiders to spider ALL the pages in my site. Here's my robots.txt:. Is this correct?
Thanking you in advance.

user-agent: mediapartners-Google*
Disallow:

7:52 am on Nov 16, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


Yes this is correct it gives mediabot and all other spider access to everything on your site
1:04 pm on Nov 26, 2004 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 13, 2003
posts:590
votes: 0


Wouldn't this:
user-agent: *
Disallow:

accomplish the same thing? You're not actually disallowing any spiders from anywhere in your site, that I can see. Or am I missing something here?

5:18 pm on Nov 26, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 7, 2003
posts:1179
votes: 0


No you are not missing anything, it's what google recommend from within the adsense login to put in your robots.txt file to ensure the pages on your site can be accessed by mediabot.

your code will work the same as will a blank robots.txt file because you are not blocking any spider from accessing your site