homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Cleaning House
blocked IP ranges may have changed

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 4625619 posted 10:19 am on Nov 24, 2013 (gmt 0)

Just a heads-up - consider authenticating all the IP ranges you have blocked. They may have changed.

I recently moved a site to a server and had reason to reevaluate my block list, which is/was huge (15 years worth.)

After weeks of work I found a significant number of ranges and/or subs now reassigned to new owners, many of which are unrelated to malicious activity ASAIK. Some ranges are now used by mobile ISP, others sliced & diced to various social networks (some beneficial for traffic... YMMV), some ranges now in the hands of ad marketing companies (important if you use Adsence or other ads services) and some ranges have either grown larger or smaller. Some ranges I had blocked now have no records at all, they've become unassigned. The changes I found are too many to list here.

So I stripped down the block lists to the usual suspects (AWS, Rackspace, Softlayer, etc..) rechecking/purging as I go.

I've also found I can get by with much less with a combo of Whitelisting, IP blocks, Head checks and UA rules. I was often blocking the same bad agent with an overkill of several methods. Cleaning house, while a definite PITA, is necessary, and for me, long overdue.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved