Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- need robots.txt for pages that are not linked?
rld3 - 6:41 am on Jan 29, 2004 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/258.htm
I'm trying to decide whether I need a robots.txt file. Does the google bot crawl pages that are in the same directory as index.html but that are not linked to index.html?
Brought to you by WebmasterWorld: http://www.webmasterworld.com