- - Search Engines
- -- Sitemaps, Meta Data, and robots.txt
- ---- Robots.txt Question?
cfel2000 - 4:57 pm on Mar 11, 2002 (gmt 0)
What do you mean by 'checking for pages which are no longer there'? A robot.txt file doesn't contain a list of pages to your site. It just tells the spider which pages you don't want crawled.
Thread source: http://www.webmasterworld.com/robots_txt/69.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com