Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- "User-Agent"?


jdMorgan - 2:36 am on Jan 22, 2003 (gmt 0)


Also, the reference cited above is for HTTP/1.1 headers, not the Standard for Robot Exclusion [robotstxt.org].

Jim
<edit> added link</edit>


Thread source:: http://www.webmasterworld.com/robots_txt/152.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com