Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Bots using robots
cooldoug - 1:29 am on Mar 9, 2005 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/567.htm
Do bots and robots really have to read the robots.txt, or can they go straight to the index.html and stard indexing on the search engine?
Brought to you by WebmasterWorld: http://www.webmasterworld.com