mass banning spiders

I am considering blocking all spiders...

[han solo]

Does anyone have any experience with mass banning or blocking spiders that seem to have no use in the server other than to take up bandwidth?

I've been having trouble with load balancing, with my new system, and I have been thinking, DIIbot, and others, don't work for search engines, so why should I let them have anything?

Any comments, shared stories, or lies :) would be greatly appreciated. Thanks all!

Cheers,
Han_Solo

[PeteU]

Most efficient way to block unwanted spiders
is to put following lines in your /etc/httpd/conf/access.conf file

order allow,deny
allow from all
deny from xxx.xxx.xxx.xxx
deny from xxx.xxx.xxx.

replace x with IP numbers of offenders
notice how you can block a whole C-class range in the second deny example
this will block them server wide for all domains hosted on the machine.

another good way of doing it is to add these lines into
VirtualHost declarations in your /etc/httpd/conf/httpd.conf file

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*DIIbot.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*BandwidthWaster.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SomeUselessSpider.*
RewriteRule ^(.*)$ - [F]

This will work on per domain basis and is also very efficient

Both of above methods can be used in your .htaccess file
but take up somewhat more of server resources, still very helpful if you don't have server admin access

[han solo]

Thanks Pete. I actually didn't need any help with the technical aspect of it, I was thinking more along the lines of potential reprecussions on my pages and clients.

Have you had any experience with this? Although I do appreciate the clear explanation of how to work with, that was Apache mod re_write, right?

Do spam harvesters, etc. tend to just start trying harder to get into a system after their blocked, thinking that they must have found something good? If anyone else cares to share their thoughts, I'd appreciate it. Thanks!

Cheers,
Han Solo