Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: phranque
I have a very large reference site and bad crawlers are huge problem. I have some automatic IP blockers, but a reliable list would definitely help in this war.
Also, having a list of "good" crawlers could allow for easing of filtering rules for legitimate search engines.
I know that Yahoo and Google share a block list, but I don't know if it is public and where to find it
joined:Sept 26, 2001
Be careful of just cut'n pasting without knowing who/what you're blocking.