User-agent: ...

[gettin there]

Evening All,

Just a quick question... I notice that many robots.txt files will have something like this:

User-agent: *
Disallow: /cgi-bin/
Disallow: /secure/

User-agent: googlebot
Disallow: /cgi-bin/
Disallow: /secure/

User-agent: slurp
Disallow: /cgi-bin/
Disallow: /secure/

My question is: What is the point of listing the user-agent when * is already specified? Is this necessary or just being used as a backup?

Thanks in advance!

[bcolflesh]

Is this necessary...

No

...or just being used as a backup?

Paranoia ;)

[Sanenet]

www.robotstxt.org

Trouble is, many people don't bother to read it properly ;)