Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Synapse user-agent


dstiles - 8:53 pm on Jan 15, 2013 (gmt 0)


A significant percentage of NEW IP ranges I've recently blocked have been blocked on this UA. Although I've flagged a few as being "servers" that was often a subjective call and may have been erroneous (but from a "bad" neighbourhoods so entails prejudice). The majority are from broadband IPs.

I'm not sure if the hits are from compromised machines (ie part of a botnet) or are part of some browser add-on/plug-in. If the latter then the result for the user is abysmal - at least from my sites. I cannot say I've seen any real evidence of compromised machines.

Since it appears to be apache-based and relatively few users have apache installed on "home" machines, and few linux machines are susceptible to compromises anyway, I'm still inclined to think this is a deliberate use of a versatile tool, possibly (probably?) with the aim of site-scraping.

From a limited number of checks in actual site logs it seems that each hit is unique - there are no other hits for the offending IP and no further attempt to hit the site from that IP - at least, not on the same day.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4532904.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com