-- Search Engine Spider and User Agent Identification
---- .htaccess BadBot Blocker
incrediBILL - 9:44 pm on Aug 1, 2013 (gmt 0)
While I agree with you in principle, never stop people from sharing or trying to learn as the risks are just as valuable to learn as the rewards from sharing such lists. Now that you've pointed out the risks, which would probably best be shared in this forum in an FAQ that I've been working on, let's examine the rewards.
While I find that block lists are problematic, I think the OP's block list has some interesting techniques that should be explored in detail IMO as most don't fully understand what's going on with the Apache code being used.
This is quite clever and potentially speeds up the scanning vs. big long monolithic lists.
This line only parses when a "w" is found, then the subset of user agents starting with "web" are processed and the various user agents starting with "web" such as webaccount, webcopier, web client, etc. A little harder for the Apache novice to read but a cool way to compact those rules.
Somewhat of a crossover with the Apache forum, but I'd like to see people explore these techniques and learn how to use them.