Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
Brett, any idea why they still spider banned pages? I would think it would be a waste of time and bandwidth for them. I recently added a number of pages to the banned list and was hoping they would not spider them anymore. Well, as long as they don't index them, that's what counts.
I had assumed that not getting spidered is the only indication that a site is banned.
Son_House, how do you know that the pages are banned?
The other issue I've seen is that pages that have no inbound links aren't listed in the directory, but are still spidered. As soon as they get inbound links, then they're added to the database.
On another note, it is interesting to block a nonexistent directory with a robots.txt and watch some spiders try to spider it. You can really spot who isn't playing fair that way.