lucy24 - 1:37 am on Feb 26, 2012 (gmt 0)
Very very few spiders react to a 403. It's like a routine shopping list: if the store is out of one item, you don't head straight for the checkout and abandon the rest of the list. Hey, who knows, maybe they'll let me in to the 84th page I ask for.
In fact I've only met one or two robots who seemed to behave differently when they didn't get the expected 403 --for example by arriving from an IP I didn't know about. And I have to say: those are scary. Better to be hit with a bunch of stupid robots than one with a brain. ("Store's out of eggplant, so don't bother about the next six items, which are all the other ingredients for eggplant parmigiana.")