Bad Bot Blackhole - More Effective Than I Anticipated

Have some questions about the results...

[SerpsGuy]

Recently I implemented a bad bot blackhole similar to the one listed on perishablepress. (http://perishablepress.com/blackhole-bad-bots/)

A quickie bg on my site: Been negative seo spammed and had my content stolen and pasted on thousands of sites. Over 1 million backlinks all spam. The latest google update restored my site from what I believed to be its final resting place in the sandbox.

Then, like clockwork, the spam started all over again. So, I decided to implement a bad bot blackhole- which I finished implementing with cloudflare and some special tweaks three days ago.

To date, I have blocked over 340 ip addresses. I was initially concerned I was blocking legitimate visitors. But after checking about 30 ips and them all turning up listed in various spam lists I decided it was working properly.

Some interesting things I found were one, nearly every user agent that has been blocked is this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0

Seriously - almost every one. Do you guys have any idea why that would be? Secondly - I would like a way to bulk loopup all these IPs that have violated my robots.txt - just to see what percentage of them are blacklisted.

Anyone have any tools or sites that can do this for free? Looking them up one by one is painfully slow and inefficient.

I highly recommend everyone implement some form of the bad bot blackhole. Especially if you have spam issues.

[lucy24]

Do you guys have any idea why that would be?

Your robot hunted for a plausible human UA at the time it was first programmed, and that's what it came up with.
:: detour to logs ::
The vast majority of Mac 10.8 + FF 21 combinations are robots; the only humans I find were from the summer of 2013. So that's your robot's approximate birthdate.

To date, I have blocked over 340 ip addresses.

I hope you don't mean exact addresses, like 12.34.56.78. Learn what the ranges are, and block those. It generally isn't worth blocking individual human IPs-- as from infected machines-- unless someone becomes especially vexatious.

If you don't already follow the Server Farms threads [webmasterworld.com] in the SSID subforum, go back and read the last year or two.

[not2easy]

There is an entire forum here dedicated to sharing this information. If you are blocking IPs individually, that is not very efficient as they usually are coming from server farms. Blocking by known groups is quite effective. The forum here is at: [webmasterworld.com...] and the Charter and Library in that forum have various situations covered along with the solutions.

BTW, the site you mentioned is not known for accurate and correct information, though it is a popular place to grab code and then bring it here to fix it. Just sayin'.