Forum Moderators: phranque
On the other hand, if they're simply scanning/scraping the Web for the majority of their information, then you may want to consider blocking the IP address ranges of *their* server providers.
However, this takes a lot of research, and can result in a huge access-control list -- I've seen some that bloated a .htaccess file to 100kB and more, even when tightly-coded. Plus, there's always the danger of blocking your own 'linking partners' and search engine thumbnailing services (as used by Ask, for example), or other servers which you may deem to have a legitimate reason to fetch pages from your site.
Jim
After several hundred pings one would pretty much have the full list of all the domains hosted on that server.
I pointed the 'anomaly' out to the good folks there and they soon had it squared away.
Jonesy