Welcome to WebmasterWorld Guest from 184.108.40.206
On important data, I run some code that stores how many times/areas an ip has hit, if there is more than N hits to a specific area/target, it can either throttle them (warning that they're hitting the site too many times in that area, and if the warning is ignored, they are automatically banned [crawlers don't read error msgs]), or silently redirect from the intended page back to the main index.
There are some massive long threads on this
[webmasterworld.com...] (and it's [b]three[b] predecessors (links in first post of each).
There is a more recent one around, but I can't find it at the moment.