Welcome to WebmasterWorld Guest from 23.22.220.37

Forum Moderators: goodroi

Message Too Old, No Replies

User-agent: ...

     
2:28 am on Dec 1, 2004 (gmt 0)

New User

10+ Year Member

joined:Nov 30, 2004
posts:6
votes: 0


Evening All,

Just a quick question... I notice that many robots.txt files will have something like this:

User-agent: *
Disallow: /cgi-bin/
Disallow: /secure/

User-agent: googlebot
Disallow: /cgi-bin/
Disallow: /secure/

User-agent: slurp
Disallow: /cgi-bin/
Disallow: /secure/

My question is: What is the point of listing the user-agent when * is already specified? Is this necessary or just being used as a backup?

Thanks in advance!

2:53 pm on Dec 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


Is this necessary...

No

...or just being used as a backup?

Paranoia ;)

3:06 pm on Dec 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


www.robotstxt.org

Trouble is, many people don't bother to read it properly ;)