e-collector Spider

Forum Moderators: open

Message Too Old, No Replies

12:26 am on Dec 27, 2004 (gmt 0)

How do I stop this thing? It doesnt seem to respect robots.txt - anyone have insight?

9:20 pm on Dec 27, 2004 (gmt 0)

On Apache, you can ban it by user-agent or IP address using mod_rewrite.

Jim

9:31 pm on Dec 27, 2004 (gmt 0)

its being used through a proxy andusing many IPs

10:08 pm on Dec 27, 2004 (gmt 0)

Then check the %{X_HTTP_FORWARDED_FOR} server variable to catch that.

If it always uses the same user-agent string, then you don't need that complication; Just block it by UA.

Jim