Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Spiders get / and /robots.txt but no more!


jimbeetle - 7:02 pm on Dec 11, 2003 (gmt 0)


I can't see anything that should stop them. Your file validates in Brett's robots.txt validator [searchengineworld.com]. The only thing might be to insert a space between the # and actual comment, not sure how nitpicky some spiders are, but that's the format used at robotstxt.org.

There doesn't look to be anything on your index page to stop them going further. You might try putting up a fresh link or two on the page and see if they follow those.


Thread source:: http://www.webmasterworld.com/robots_txt/217.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com