chiyo - 3:28 pm on Jan 12, 2003 (gmt 0)

tristan. are you absolutely sure? It seems to go against common understanding of robot.txt functionality and specification. We have robotted txtsed out our cgi-bin for years and dont see any evidence that this directory was being followed by Google or any robots that respect robots.txt.

Robots.txt is only advisory. Its still open for any spider to spider it, and you know what some people are like! Are you sure that this was just the case with a rogue spider from an email crawler or the like?