Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
Several search engines support various 'extensions' to the robots.txt protocol. Webmasters must take care that these proprietary extensions are only used in robots.txt policy records which apply to those specific robots that support them.
The effects of using a wild-card URL-path in a policy record for a robot that doesn't understand wild-cards might range from 'no effect' to 'disastrous'.
'protocol' is de-facto what people call it; it does not have any associated RFC:
robotstxt.org was born as a supporting website for (closed now) email@example.com mailing list; their database and info is extremely outdated.