Blackspider?

recently showed up

[mcneely]

Blackspider

Showed up after installing a few weblogs of late.

[wilderness]

Was it from an APNIC range?

# "keep_out" or what ever term you use
SetEnvIfNoCase User-Agent Spider keep_out

[mcneely]

RewriteCond %{HTTP_USER_AGENT} ^blackspider [OR]

[wilderness]

RewriteCond %{HTTP_USER_AGENT} ^blackspider [OR]

believe the following would be more effective and require less long-term redundancy:

RewriteCond %{HTTP_USER_AGENT} ^spider [OR,NC]

Is their a legitimate SE besides Lycos that utilizes the term "spider"?

[Mokita]

Blackspider belongs to Websense:

[webmasterworld.com...]

Banning by user agent is definitely the best way to go, as they are using dynamic dial-up IPs.

[blend27]

Mokita, I've must of missed something but why would one want to block Websense spider?

[wilderness]

why would one want to block Websense spider?

Past practices!

[keyplyr]

I was under the impression this blackspider was a content filter bot sent per event.

[mcneely]

Websense still comes in to our sites like it should, only under their "own" range(s) and without the use of this silly UA.

Websense still comes out of San Diego just like always, but the Blackspider was coming in from across the pond. Pulling out of RIPE for most of their IP's and running under mailcontrol.com (currently redirecting to websense).
Surfcontrol is a Euro entity, handling Euro monitoring. What a Euro monitoring system is doing roaming the States is beyond me. San Diego (Websense) remains active and is free to come and go as it will.

The blackspider ranges, however, were all over the board as I recall, so I gave it a boot to the head.

Blackspider was a crawler for Surfcontrol (Berkshire)

Incidently;

There wasn't any direct Euro traffic either before or after the appearance of Blackspider.

Thought that since Websense builds their own parsing and monitoring software, that after Surfcontrol was purchased by Websense, that a disgruntled mate took the code home with him to run as he saw fit to do

Don't laugh, it wouldn't be the first time technology like this was "borrowed" from a defunked company like Surfcontrol.
It could also easily explain why the IP's didn't match up.

[keyplyr]

Since it's a user event filter, then yes it *would* come from all IP ranges. Anyone can use it from the company, their server, et al...

[Hobbs]

Is their a legitimate SE besides Lycos that utilizes the term "spider"?

Searching my logs for "spider" for the past 8 days:

Sogou web spider/3.0 (banned)
sogou develop spider (banned)
Sogou Orion spider (banned)
YodaoBot (banned)
Gigabot/3.0 (banned)
LTI/LemurProject Nutch Spider/Nutch-1.0-dev (banned)

Allowed:

Baiduspider
Speedy Spider entireweb
webconfs search-engine-spider-simulator
INGRID/2.0 spsearch.ilse.nl Startpagina dochter links spider

any of those worth allowing?

[wilderness]

any of those worth allowing?

NARY a one.

Nutch anything is a throw as well!
"link" anything a toss too.