Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt


Son_House - 12:57 am on Jul 5, 2001 (gmt 0)


> Nor will it stop them from spidering the pages banned by the robots.

Brett, any idea why they still spider banned pages? I would think it would be a waste of time and bandwidth for them. I recently added a number of pages to the banned list and was hoping they would not spider them anymore. Well, as long as they don't index them, that's what counts.


Thread source:: http://www.webmasterworld.com/robots_txt/156.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com