Black Widow Web Crawler

Something to watch out for?

[MarieC]

I haven't seen this on my site. Stumbled upon it by sheer accident. www.fsconsult.net/products/ (halfway down the page):

The Black Widow Web Crawler is a very high speed, targetable web crawler which allows one to crawl specific site very quickly. [more...]

Also found a short reference to it here:

[webmasterworld.com ]

[jdMorgan]

It came.
I saw.
I banned it.

(With apologies to Caesar) :)

Jim

[2oddSox]

I've used this tool a number of times (with legitimate reason ;) - probably the scariest thing about it is the amount of information that it gathers that can then be used for 'identity theft' using other means. Sometimes the most innocuous information left open on a website can be a launching pad to something more extreme.

2odd...

[wilderness]

Jim,
You have a UA?
TIA
Don

[jdMorgan]

Don,

I'm using "RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]" for this one. And then there's the spider-trap script in addition to that UA block.

Jim