Welcome to WebmasterWorld Guest from 54.145.173.147

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

mass banning spiders

I am considering blocking all spiders...

   
9:50 pm on Nov 15, 2000 (gmt 0)

10+ Year Member



Does anyone have any experience with mass banning or blocking spiders that seem to have no use in the server other than to take up bandwidth?

I've been having trouble with load balancing, with my new system, and I have been thinking, DIIbot, and others, don't work for search engines, so why should I let them have anything?

Any comments, shared stories, or lies :) would be greatly appreciated. Thanks all!

Cheers,
Han_Solo

2:21 am on Nov 16, 2000 (gmt 0)

10+ Year Member



Most efficient way to block unwanted spiders
is to put following lines in your /etc/httpd/conf/access.conf file

order allow,deny
allow from all
deny from xxx.xxx.xxx.xxx
deny from xxx.xxx.xxx.

replace x with IP numbers of offenders
notice how you can block a whole C-class range in the second deny example
this will block them server wide for all domains hosted on the machine.

another good way of doing it is to add these lines into
VirtualHost declarations in your /etc/httpd/conf/httpd.conf file

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^.*DIIbot.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*BandwidthWaster.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^.*SomeUselessSpider.*
RewriteRule ^(.*)$ - [F]

This will work on per domain basis and is also very efficient

Both of above methods can be used in your .htaccess file
but take up somewhat more of server resources, still very helpful if you don't have server admin access

2:28 pm on Nov 16, 2000 (gmt 0)

10+ Year Member



Thanks Pete. I actually didn't need any help with the technical aspect of it, I was thinking more along the lines of potential reprecussions on my pages and clients.

Have you had any experience with this? Although I do appreciate the clear explanation of how to work with, that was Apache mod re_write, right?

Do spam harvesters, etc. tend to just start trying harder to get into a system after their blocked, thinking that they must have found something good? If anyone else cares to share their thoughts, I'd appreciate it. Thanks!

Cheers,
Han Solo