Has the number of new crawlers and things being launched on the web slowed down?
I'm not noticing nearly as much stuff as I did a couple of months ago so either I'm not looking hard enough or more things are in stealth than ever before.
keyplyr
11:45 pm on Jun 9, 2010 (gmt 0)
At one time I had over 300 UAs blocked along with 30 IPs and a few other assorted rules.
Nowadays, I white list a half dozen UAs validated by rDNS, block only a dozen UAs categorically by name, and block approx 80 IP ranges. My htaccess file is 1/4 the size it used to be and maintenance is much easier.
Are there less new crawlers being launched? Possibly, but I'm not spending nearly as much effort/focus on them any longer so it's difficult to know.
enigma1
12:46 pm on Jun 15, 2010 (gmt 0)
Try to cross-reference with what is happening with the search engines. Usually many crawlers copy popular SEs (even SEs copy each other) so if the popular spiders change things it will affect other crawlers I think.
One of the things I recently did (6 months ago) was to block access to one of the major SEs on a site that's online for many years. The number of bad bots and weird UAs, dropped significantly as I can see from the logs, of course along with the traffic.
blend27
12:20 am on Jun 16, 2010 (gmt 0)
incrediBILL, maybe they just figured out which sites are yours and left you along?
Or maybe ElasticCloud changed their AUP(if not they should). I got over 1000 Colo/Hosting IP Ranges blocked and really don't monitor those as I used to a year back or so, but You right it's been hush-hush for the past couple of month.
