Do you guys ban Cyveillance?

They don't behave very well sometimes

[kewlbeezer]

I got attacked by Cyveillance this morning (50 pages searched in less than 290 seconds -- 1 page every 6 seconds or so)... it was from 63.148.99.247 With a spoofed UA of Mozilla/4.0+(compatible;+MSIE+5.01;+Windows+NT+5.0) (the +'s are from IIS' logs).

They don't obey robots.txt, and add a lot of traffic to the server. Should I block em?

[jeremy goodrich]

I would block them, but keeping up with rogue bots is becomming more and more of a headache, the more people that decide, "hey, lets crawl the web & irritate hordes of webmasters..."

Any bot that also doesn't respect robots.txt or creates unnecessary load, even one that belongs to a search engine - if they aren't giving you something in return, I would say ban them.

For a while, to protest, I even had altavista banned from all of my sites...though I let them have it these days. :)

[jdMorgan]

You should consider installing key_master's bad-bot trap [webmasterworld.com] script. Cyveillance will ban themselves rather quickly if you install it. Then all you have to do is go through the ban list once a week or so, and look up the IP ranges of repeat offenders using related multiple IP addresses. This will help to compress your ban list and make it more efficient.

Jim

[kewlbeezer]

So that's a "yes, beezer, you should ban them" :)

Also, on IIS its kinda hard to deploy an .htaccess file :-D

[wilderness]

<snip>on IIS its kinda hard to deploy an .htaccess file</snip>
[webmasterworld.com...]

[fiestagirl]

We use IIS and for any robots like this that are easily identified and use the same IP or group of IPs get blocked at the firewall. Nice and neat. Adios robot.

[nativenewyorker]

Cyveillance (despite claims on their website) does not honor robots.txt. Their bot also does not identify itself correctly and spoofs UA strings.

Their IP address ranges to block are:

63.148.99.224-255
65.118.41.192-223

Ted

[mgream]

I have to chuckle at a business week headline that you can find if you search for cyveillance that says "Cyveillance: Helping to Keep the Web Honest". This statement is laughable as the behaviour of their UA's are clearly unethical (i.e. they don't abide to robots.txt and they use dubious user agent identification), bordering on the edge of dishonesty. They may aim to detect IP infringements, but that's no justification for using an unethical approach.

[bird]

Do you guys ban Cyveillance?

Of course, and here's why:
- They are dishonest in what they say vs. what they do.
- They eat up tons of bandwidth when left unchecked.
- They are of no benefit whatsoever to my site or my visitors.

however, since I'm a polite person, I'm not returning a 403 even to misbehaving spiders. They will simply get a standard serving of an empty page. The message of that being "yes, you're welcome, but there's nothing of interest to be found here for you".