Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt Question?
cfel2000 - 4:57 pm on Mar 11, 2002 (gmt 0)Thread source:: http://www.webmasterworld.com/robots_txt/69.htm
What do you mean by 'checking for pages which are no longer there'? A robot.txt file doesn't contain a list of pages to your site. It just tells the spider which pages you don't want crawled.
Brought to you by WebmasterWorld: http://www.webmasterworld.com