Xoc - 2:33 pm on Mar 27, 2001 (gmt 0)

The problem is that there is no definitive spec for robots.txt. As far as I know, there is no W3C or RFC spec for it. So it means whatever a search engine wants it to mean. Do search engines use wildcards? Google does. Does an earlier rule override a later one? Looksmart seems to think so. But does Google? Who knows what Google really does with Looksmart's robots.txt.

The problem is not just for spiders. How do you come up with a correct robots.txt, when the spec is ambiguous?

Edited by: Xoc