homepage Welcome to WebmasterWorld Guest from 54.242.18.232
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt-newbie needs help
Is this correct?
cybervette




msg:1526084
 3:08 am on Nov 16, 2004 (gmt 0)

HI All, What I am trying to do here is to allow google's adsense bot,for target ads, and I want to allow all other spiders to spider ALL the pages in my site. Here's my robots.txt:. Is this correct?
Thanking you in advance.

user-agent: mediapartners-Google*
Disallow:

 

ncw164x




msg:1526085
 7:52 am on Nov 16, 2004 (gmt 0)

Yes this is correct it gives mediabot and all other spider access to everything on your site

sonjay




msg:1526086
 1:04 pm on Nov 26, 2004 (gmt 0)

Wouldn't this:
user-agent: *
Disallow:

accomplish the same thing? You're not actually disallowing any spiders from anywhere in your site, that I can see. Or am I missing something here?

ncw164x




msg:1526087
 5:18 pm on Nov 26, 2004 (gmt 0)

No you are not missing anything, it's what google recommend from within the adsense login to put in your robots.txt file to ensure the pages on your site can be accessed by mediabot.

your code will work the same as will a blank robots.txt file because you are not blocking any spider from accessing your site

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved