homepage Welcome to WebmasterWorld Guest from 54.211.180.175
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
User-agent: ...
gettin there




msg:1525355
 2:28 am on Dec 1, 2004 (gmt 0)

Evening All,

Just a quick question... I notice that many robots.txt files will have something like this:

User-agent: *
Disallow: /cgi-bin/
Disallow: /secure/

User-agent: googlebot
Disallow: /cgi-bin/
Disallow: /secure/

User-agent: slurp
Disallow: /cgi-bin/
Disallow: /secure/

My question is: What is the point of listing the user-agent when * is already specified? Is this necessary or just being used as a backup?

Thanks in advance!

 

bcolflesh




msg:1525356
 2:53 pm on Dec 1, 2004 (gmt 0)

Is this necessary...

No

...or just being used as a backup?

Paranoia ;)

Sanenet




msg:1525357
 3:06 pm on Dec 1, 2004 (gmt 0)

www.robotstxt.org

Trouble is, many people don't bother to read it properly ;)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved