Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- "User-Agent"?

Dreamquick - 11:48 pm on Jan 21, 2003 (gmt 0)


FWIW I'm running a robots.txt without quotes around the "user-agent" prefix and everything seems to be obeying me
e.g.

User-agent: ia_archiver
Disallow: /

- Tony

Thread source:: http://www.webmasterworld.com/robots_txt/152.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com