Welcome to WebmasterWorld Guest from

Forum Moderators: phranque

Message Too Old, No Replies

IP block list

shared list of bad and good crawlers



7:08 pm on Jun 20, 2004 (gmt 0)

10+ Year Member

Where do I find some reliable and updatable lists of "bad" and legitimate crawlers?

I have a very large reference site and bad crawlers are huge problem. I have some automatic IP blockers, but a reliable list would definitely help in this war.

Also, having a list of "good" crawlers could allow for easing of filtering rules for legitimate search engines.

I know that Yahoo and Google share a block list, but I don't know if it is public and where to find it


6:27 am on Jun 21, 2004 (gmt 0)

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

One size doesn't really fit all when it comes to what agents you wish to allow, however this thread may give you an idea A Next to Perfect Ban List - Part 3 [webmasterworld.com].

Be careful of just cut'n pasting without knowing who/what you're blocking.


Featured Threads

Hot Threads This Week

Hot Threads This Month