Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Do we need robots.txt?


pageoneresults - 8:40 am on Jun 10, 2002 (gmt 0)


User-agent: *
Disallow:

> I wouldn't recommend it. There are some spiders that will incorrectly interpret that as blocking all content.

Ouch! I guess I need to rewrite all my instructions on creating a robots text file. Brett, do you know which spiders misinterpret the above?


Thread source:: http://www.webmasterworld.com/robots_txt/112.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com