Page is a not externally linkable
Son_House - 12:57 am on Jul 5, 2001 (gmt 0)
Brett, any idea why they still spider banned pages? I would think it would be a waste of time and bandwidth for them. I recently added a number of pages to the banned list and was hoping they would not spider them anymore. Well, as long as they don't index them, that's what counts.
> Nor will it stop them from spidering the pages banned by the robots.