Robot snooping

[toolman]

I just got robbed! A competing company just slurped thru my entire site.

It was very polite at 1 request a second and didn't carry a spider "footprint". All I have is an ip and a UA of Mozilla/5.0.

I'm already blocking UA's by mod_rewrite. The question is...how can I stop something like this. Is there a regular expression that would id Mozilla/5.0 and requests in excess of 10@1 per sec?

[Brett_Tabke]

Offhand toolman, the only way to work it, is to session track. That can be fairly involved and resource hungry.

[littleman]

Basically, there is no way to block someone who wants to get at your publicly available stuff if they know what they are doing.