jdMorgan - 2:33 am on Jan 22, 2003 (gmt 0)

The error message in the Search Engine World robots.txt checker is somewhat ambiguous. It is telling you to not capitalize the "A" in the word "Agent", not that you must quote the name of the user-agent.

The error message reads:
ERROR Field names are case sensitive. It should be written as: "User-agent" as written in the standard.

This may not be in compliance with the RFC, but it is what the robots.txt checker is calling for.

Note also that in the "listing" that the checker produces, it will "correct" this capitalization and show "User-agent: psbot" for example, even though the source file says, "User-Agent: psbot". It is not showing you your source file, but rather a modified version of it.

I was able to get a test version of robots.txt to pass with or without white space after User-agent:

White space after User-agent: and Disallow: is optional, but it does improve readability for humans.

The robots.txt checker script is not perfect, but it does help to eliminate most errors.

Jim