Sgt_Kickaxe - 9:23 pm on Dec 10, 2012 (gmt 0)
I've been managing an htaccess blacklist of bots with varying degrees of success, unfortunately I tend to add a bot after it has done the deed so to speak. Lists like the one on askapache helped start the list but it has evolved over time. [askapache.com...]
I also manage a whitelist of bots in robots.txt though most bad bots don't bother reading it much less complying.
I've decided that I want to add a few reputable bots to my htaccess blacklist because I've caught them ignoring my robots.txt. Not only that but it dawned on me that these "reputable SEO data gathering resellers" are selling my stats to my competitors.
I'm ready to toss the entire range of SEO bots to the curb but I only know of a few. example: ezoom (seomoz), ahrefsbot and sitebot don't bring me traffic but they get my stats sold on services I may not be using for this site.
Is there a complete list of known SEO data gathering site bots? Also, since they may not always declare themselves, is there a list of IP ranges for these types of data reseller SEO sites?
I love the services, and the companies behind them, but aggregating and selling my stats to my competitors? Pass.