- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- need robots.txt for pages that are not linked?
rld3 - 6:41 am on Jan 29, 2004 (gmt 0)
I'm trying to decide whether I need a robots.txt file. Does the google bot crawl pages that are in the same directory as index.html but that are not linked to index.html?
Thread source: http://www.webmasterworld.com/robots_txt/258.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com