Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Is this enough


trillianjedi - 11:19 am on Feb 2, 2004 (gmt 0)


All I want is for every robot for every SE and every directory to visit my website.

You can safely just remove your robots.txt file completely in that case.

It's only used for listing search engine spiders that you do *not* want indexing your site.

TJ


Thread source:: http://www.webmasterworld.com/robots_txt/264.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com