Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- DomainCrawler


zeus - 11:18 pm on Nov 5, 2008 (gmt 0)


lets make a list of all those #*$!rapers

right now im blocking

<Limit GET HEAD POST>
order allow,deny
deny from 216.145.16.0/24
deny from 66.249.16.*
deny from 66.249.17.*
deny from 64.246.165.*
deny from 64.79.192.0/19
deny from 64.246.165.128/25
deny from 66.249.0.0/19
deny from 209.59.192.0/19
deny from 216.145.16.0/24
deny from 83.168.240.5
deny from 66.249.0.0/19
allow from all
</LIMIT>

but maybe we should place a name also and is there a option to block those on whole server, instead changing htaccess on each domain


Thread source:: http://www.webmasterworld.com/search_engine_spiders/3768838.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com