Robot exclusion protocol..

[skirril]

There seem to be two standards:
a) currenlty a de facto std, dates from 1994:

User-agent: foo
Disallow: /bar
Disallow: /baz

User-agent: bar
(idem)

-no Allow clause
-one ua per line
#this is a comment
- unrecognised fields are ignored

[robotstxt.org]

b) This is an proposed RFC, not a formal standard. New Syntax:

User-agent: foo
User-agent: bar
Allow: /
Disallow: /bar
Disallow: /baz

User-Agent: baz ..

[robotstxt.org]

[wilderness]

<snip>This is an proposed RFC
Date of proposal submission
Dec 4, 1996

Date of proposal expiration.
Expires June 4, 1997

For anything to be expired and pending after nearly FIVE years in todays fast paced internet world is hardly reflective of what is in the cards for the future.

However I hope I'm wrong and portions of the proposed robots exclusion are implemented tomorrow.
Especially the use of trailing slases as related to file controls.