- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- "User-Agent"?
bcc1234 - 2:06 am on Jan 22, 2003 (gmt 0)
|That may be because "User-Agent" is incorrect. It must be "User-agent" - No capital "A" in "Agent." |
Not sure about that. HTTP RFC states it as User-Agent and most software treats it the same regardless of the letter case.
I would assume robots.txt would rely on the RFC.
Thread source: http://www.webmasterworld.com/robots_txt/152.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com