Welcome to WebmasterWorld Guest from 3.80.4.76

Forum Moderators: goodroi

Message Too Old, No Replies

Bad Bot Blackhole - More Effective Than I Anticipated

Have some questions about the results...

     
6:39 am on Dec 19, 2014 (gmt 0)

Junior Member

5+ Year Member

joined:Mar 9, 2013
posts:115
votes: 1


Recently I implemented a bad bot blackhole similar to the one listed on perishablepress. (http://perishablepress.com/blackhole-bad-bots/)

A quickie bg on my site: Been negative seo spammed and had my content stolen and pasted on thousands of sites. Over 1 million backlinks all spam. The latest google update restored my site from what I believed to be its final resting place in the sandbox.

Then, like clockwork, the spam started all over again. So, I decided to implement a bad bot blackhole- which I finished implementing with cloudflare and some special tweaks three days ago.

To date, I have blocked over 340 ip addresses. I was initially concerned I was blocking legitimate visitors. But after checking about 30 ips and them all turning up listed in various spam lists I decided it was working properly.

Some interesting things I found were one, nearly every user agent that has been blocked is this:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0

Seriously - almost every one. Do you guys have any idea why that would be? Secondly - I would like a way to bulk loopup all these IPs that have violated my robots.txt - just to see what percentage of them are blacklisted.

Anyone have any tools or sites that can do this for free? Looking them up one by one is painfully slow and inefficient.

I highly recommend everyone implement some form of the bad bot blackhole. Especially if you have spam issues.
7:05 am on Dec 19, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15818
votes: 853


Do you guys have any idea why that would be?

Your robot hunted for a plausible human UA at the time it was first programmed, and that's what it came up with.
:: detour to logs ::
The vast majority of Mac 10.8 + FF 21 combinations are robots; the only humans I find were from the summer of 2013. So that's your robot's approximate birthdate.

To date, I have blocked over 340 ip addresses.

I hope you don't mean exact addresses, like 12.34.56.78. Learn what the ranges are, and block those. It generally isn't worth blocking individual human IPs-- as from infected machines-- unless someone becomes especially vexatious.

If you don't already follow the Server Farms threads [webmasterworld.com] in the SSID subforum, go back and read the last year or two.
7:10 am on Dec 19, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4469
votes: 332


There is an entire forum here dedicated to sharing this information. If you are blocking IPs individually, that is not very efficient as they usually are coming from server farms. Blocking by known groups is quite effective. The forum here is at: [webmasterworld.com...] and the Charter and Library in that forum have various situations covered along with the solutions.

BTW, the site you mentioned is not known for accurate and correct information, though it is a popular place to grab code and then bring it here to fix it. Just sayin'.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members