|w3who.net's hidden crawler, crouching user agent|
Who is w3who.net? Another domain intel site.
Appears to maybe use 3rd party data with some direct updates of it's own.
IP: 188.8.131.52 (host and crawler)
USER AGENT: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.1 (KHTML, like Gecko) Chrome/6.0.428.0 Safari/534.1
Definitely trying to fly under the radar, too bad.
IP RANGE: 184.108.40.206 - 220.127.116.11
These are the Linode ranges I block. Ever since they started their Cloud Servers there's been increasing trouble from every one of these ranges.
18.104.22.168 - 22.214.171.124
126.96.36.199 - 188.8.131.52
184.108.40.206 - 220.127.116.11
18.104.22.168 - 22.214.171.124
126.96.36.199 - 188.8.131.52
What we need is a nice big list of IPs and IP range lists to pin to the top of the forum for everyone to share.
Problem is I can't use hard firewall blocks otherwise I'd never figure out where these crawlers are using the data.
Someone has to make the sacrifice! ;)
But it [big list of IPs] would likely end up just like the big lists of UAs that used to get posted a couple years ago. Trouble is, what's bad for one web site is welcomed at another.
Example: Many of us block all the social parasites (twitter bots, linkedin, facebook, et al) but some webmasters thrive on their data being syndicated with the help of these harvesters.
[edited by: keyplyr at 7:33 am (utc) on Nov 6, 2011]
|Example: Many of us block all the social parasites (twitter bots, linkedin, facebook, et al) but some webmasters thrive on their data being syndicated with the help of these harvesters. |
I really don't have a problem with how people use the list either way.
My issue is validation, as long as it can be validated, it's better for everyone.
Those user agent lists without IP context are useless, and those were blacklists, again useless IMO unless you use them to build whitelists! hehe.
FWIW, I actually let facebook into my site because their security system does validate links are OK for facebook users to visit, which IMO is a good thing.
My linode list is:
Bill - I have a list of over 3800 "server" ranges of which probably 2500 or so are true server farms. Others are either invasive statics or me mis-interpreting the signs or even, in the case of RU and UA, applying a vindictiveness I probably shouldn't after a particuarly bad batch of /21 /22 /23 and /24 ranges.
I would offer the list but I feel it's too mis-leading and, in part, down to personal prejudice.