jdMorgan - 11:31 pm on Jun 14, 2003 (gmt 0)

Robots.txt is useful to disallow specific robots from specific pages/scripts/images, etc. Like your meta-description (in some engines), it gives you additional control over the "presentation" of your site, allows you to prevent visiotrs from entering into your site on some random page, etc.

Disallowing pest 'bots is a secondary function - although it is a much-discussed subject.

A better example robots.txt might be:

User-aqent: Googlebot
User-agent: Slurp
User-agent: Scooter
Disallow: /shopping_cart/
Disallow: /cgi-bin/
Disallow: /mail/e-mailform.html

Ref: A Standard for Robots Exclusion [robotstxt.org]

HTH,
Jim