Secure and other valid proxy sources

An escape from servers

[dstiles]

As may be obvious from some of my recent postings, I've been trying to improve the traffic for my clients by finding more "proxy" IPs. It began when one client complained of losing orders, although my actual opinion on that was summed up mostly, but not entirely, by "seasonal".

Over the past week-ish I've unblocked scansafe, ironport, websense, blackberry, opera, webroot, silk and flipboard within Amazon and similar: services I've blocked either because they hide within server farm ranges or their activity appeared in the past to be bot-like (there are more but I forgot to make notes!). There are also ones I already had cleared such as synetrix and research machines for UK education.

I've just discovered that zscaler is not only a secure proxy but has hundreds of small IP ranges. The only one I've found so far is within an internap range: 77.242.202.224/27. I say the only one: I recall the name going some way back but it was always a sub-range of a more popular server farm, which I then (ignorantly) blocked.

1. Does anyone have a reasonable list of zscaler proxy IPs?

2. I have evidence of at least one proxy IP within an otherwise-blocked MS range. Does anyone have a list? The sample one did not show up in DNS with any reasonable rDNS name so running a lookup on all IPs is probably not useful.

3. What other proxy ranges should I be looking at?

4. Are there similar in-server ranges that send valid traffic (eg Nokia within Amazon).

If there is any interest in this I will attempt to disentagle what I already have, assigning numbers to services.

[lucy24]

Sublets is why they are broken into small ranges. Maybe not a good idea to combine them.

I looked it up. Currently 185.25.84-85 and 86 claims to be Privax Houston, while 185.25.87 calls itself Privax Madrid. I think it's safe to lump them together. (Further brief detour confirms that Privax and CMA, er, HMA are the same people.)

[keyplyr]

Good to know thanks, but my statement was geared more to these small proxy ranges not having a history of being permanent. Seen a few change frequently, especially the small proxies. I get the impression that larger assignments may slice-off smaller ranges and lease them out for various reasons, often as proxy, but temporary until they're re-purposed for business needs.

[keyplyr]

IP: 209.85.238.71
Hostname: rate-limited-proxy-209-85-238-71.google.com
ISP: Google

@aristotle - I don't block it, or any other Google proxy. I get big mobile traffic. I wouldn't if I categorically blocked Amazon & Google servers, both of which are home to lots of mobile app developers:

Google Proxy
209.85.128.0/17
209.85.128.0 - 209.85.255.255

Just like any other range, I take a surgical approach... blocking the bad & allowing the beneficial. Takes a lot of oversight, but that's the job.

[trintragula]

(Dunno about anyone else, but I get annoyed when I hear about anything even remotely suspect coming from 185. Oi! You know there are real people desperate for an IP block over there.)

Here's what I had from 185./8 today:

ip              n  blocked  useragent                                                                                             
185.31.210.nnn  3        3                                                                                                        
185.26.92.nnn   4        4  GarlikCrawler/1.2 (http: //garlik.com/, [e-mail])                                           
185.3.35.nnn    6        6  Mozilla/4.0 (compatible; Synapse)                                                                     
185.20.4.nnn    1        1  Mozilla/5.0 (TweetmemeBot/4.0; +http: //datasift.com/bot.html) Gecko/20100101 Firefox/31.0            
185.13.37.nnn   3        3  Mozilla/5.0 (Windows NT 6.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.71 Safari/537.36 
185.31.136.nnn  8        8  NetLyzer FastProbe                                                                                    
185.18.61.nnn   8        0  Opera/9.80 (Windows NT 5.1) Presto/2.12.388 Version/12.17  

Germany, UK, Russia, Finland, Croatia, France (not in that order).

So only two browser-like useragents that stand any chance of being admitted.

• Chrome/39.0.2171.71 appears to be a spambot botnet, that the blocker seems to have under control. Six registration attempts - all cut short. There's some unsavoury referer spam with it just for good measure. It was visiting through a Tor exit too - which is not usually a good sign, but not in itself grounds for blocking on my site.
  
• The Opera browser at the end is from a telecom provider, using an Opera-owned proxy and appears to be human, even though that useragent is commonly used for bad purposes. A reasonable, short visit: good referers, plausible requests in a plausible order, plausibly spaced. If it was someone scraping they'd have to be incredibly patient and trying really, really hard not to get stopped. It seems to pass muster with the blocker for the moment, which is good enough for me.
  

So which country provided the good traffic? Croatia. Go figure.

[keyplyr]

packetflip.com
104.140.151.0/24
104.140.151.0 - 104.140.151.255

From the looks of their site, I would assume they have other proxy ranges. Sorry I can't look them up (I'm on the road using a mobile phone.)

[dstiles]

And what part of "Highly Anonymous U.S. HTTP Private Proxies" do you like? :)

Not for me. I like to know there is someone reasonably human behind a proxy.

[keyplyr]

And where did I say I "like" it? I don't "like" Hide My Ass either... I'm just listing information. Do with it as you will.

I'm not allowing these ranges myself, although someone else might wish to.

[aristotle]

Looks like there is another Google proxy at:

72.14.199.71

Just got a hit from it just like an earlier hit from a Google proxy at 209.85.238.71, which I posted about here in this thread on May 9.

Since it's coming from different IPs and proxies, I don't think it's a human, but instead could be some kind of anonymous bot that uses various Google proxies.

Edit: P.S. I just noticed that there was another hit earlier today from a Google proxy at 72.14.199.77. So that makes three different Google proxies that this has used.

[lucy24]

A while back I put in a rule that blocks visits from certain Google ranges if the UA doesn't contain "google" and there's no X-Forwarded-For header. I don't know if this is still the optimal approach, but I can't remember any recent human lockouts.

[keyplyr]

@aristotle - your 72.14.199.71 visit was from a known Google proxy at:

72.14.192.0/18
72.14.192.0 - 72.14.255.255

[aristotle]

Thanks Lucy and keyplyr -- Actually I don't understand what Google uses these proxies for, or why they even created them. That's one reason why I haven't tried to do any kind of blocking on them. Maybe someone can explain exactly what their purpose is.

[bhukkel]

I see two types of Google proxies in my logs:

rate-limited-proxy-x-x-x-x.google.com, is used by the Mediapartners-Google crawler (adsense)
google-proxy-x-x-x-x.google.com, is used by the Chrome-Compression-Proxy service. A service to speed up mobile browsing and used by Chrome for mobiles

[keyplyr]

I don't understand what Google uses these proxies for...Maybe someone can explain exactly what their purpose is.

From what I understand, and it's not much, these proxies can be used for anything. As for their purpose... its a business model. Control the IP ranges and you control the internet.

As bhukkel said, some of these proxy ranges are used by Google itself. Besides the 2 he listed, another is the Google Image Proxy used to grab the image from your site when someone (or you) post a link to your site at Google+. A nice image next to the incoming link helps bring visitors :)

Some other Google proxy ranges are used by app developers. Since I have a strong mobile presence, I want these ranges to have access to my server. Some people don't. Personally, I don't block any of them by range. I do block a few by UA.

Note: for years my defensive strategy was a knee-jerk reaction to block almost anything besides humans and a few SEs. I have since seen the error in that thinking.

[trintragula]

A bit more detail:

Apart from browser user agents, here are the special purpose user agents I've seen from the google-proxy style proxies:

Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0
Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google favicon
urlresolver
Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
Mozilla/5.0 (compatible; Google-Apps-Script)
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
Mozilla/5.0 (en-us) AppleWebKit/534.14 (KHTML, like Gecko; Google Wireless Transcoder) Chrome/9.0.597 Safari/534.14
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0,gzip(gfe)

The last of these is google translate doing its thing.
ggpht.com is, I think, something to do with Picasa.
The Favicon doesn't send an XFF with a different public address.
Web Preview, Transcoder and Google Translate all do (I think).

The Chrome Compression proxy is distinguished by the Via header, but passes on the browser user agent.

I see occasional visits from humans with IPv6 addresses on various mostly mobile devices. I've only ever seen them come via these Google Proxies.

So the above proxies are partly human, partly not.
Bear in mind that if the proxies are public, people can send whatever user agent they like... Google could easily police that to at least prevent them from pretending to be Google, but I don't suppose they do.


Here are those I've seen from the rate-limited-proxy style proxies.

Mediapartners-Google
AdsBot-Google (+http://www.google.com/adsbot.html)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.71 Safari/537.36
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 7 subscribers; feed-id=nnnnnnnnnnnnnnnnnnnnn)
Mozilla
Mozilla/5.0 (compatible) Feedfetcher-Google;(+http://www.google.com/feedfetcher.html)

I assume feedfetcher is an RSS reader of some kind (I do have RSS on my site).

In spite of the Chrome/28 browser UA in the middle there, these don't seem to be humans, and the proxies do not forward a client address via X-Forwarded-For.

I have seen the following IPs as google-proxy:

64.233.172.nnn
66.102.6.nnn
66.102.7.nnn
66.102.8.nnn
66.249.80.nnn
66.249.81.nnn
66.249.82.nnn
66.249.83.nnn
66.249.84.nnn
66.249.85.nnn
66.249.88.nnn
66.249.93.nnn

and the following as rate-limited-proxy:

209.85.238.nnn
66.249.89.nnn
66.249.90.nnn
66.249.91.nnn
66.249.92.nnn
72.14.199.nnn

I'm not assuming these are all /24, which is why I haven't aggregated them in CIDR format.


I've also occasionally seen private addresses, and addresses from the IPv4 reserved or multicast ranges in the X-Forwarded-For headers from google proxies. Something to watch out for.

[keyplyr]

BTW - there's no forum rule you need to use "nnn" when exemplifying company ranges (Google, Bing, Amazon, etc). That is just a privacy concern for humans on ISPs :)

[trintragula]

BTW,
Some interesting background on HMA in the press: [bbc.co.uk...]

[trintragula]

BTW - there's no forum rule you need to use "nnn" when exemplifying company ranges (Google, Bing, Amazon, etc). That is just a privacy concern for humans on ISPs happy!

Thx. I was just using 'nnn' here because I'm not completely sure what the ranges are. The rate-limited-proxies mostly don't seem to have rDNS beyond .127, so I'm not quite sure what the deal is there.

[keyplyr]

Another VPN service is PrivateInternetAccess.com:
209.222.5.224/28
209.222.5.224 - 209.222.5.239

which has its ranges inside Choopa:
209.222.0.0/19
209.222.0.0 - 209.222.31.255

I see a couple humans(?) per week coming from PIA ranges (although they could be the same users just getting different dynamic IPs) with valid looking Facebook, Google or Bing referrers. And there may be more PIA ranges I haven't seen yet. However, unlike proxies, users on these covert VPN services should know why they're being blocked.

[added]
PIA also may be using ranges from prolexic.com:
72.52.0.0/18
72.52.0.0 - 72.52.63.255

[keyplyr]

Also a VPN service, although they call themselves a proxy, is guardster.com, who use *at least* these two Linode ranges:
72.14.176.0/20
72.14.176.0 - 72.14.191.255
198.58.96.0/19
198.58.96.0 - 198.58.127.255

Example:
198.58.105.167 - - [25/May/2015:03:28:03 -0700] "GET /example.html HTTP/1.1" 403 1531 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"

[trintragula]

I've seen a few hits from Bluecoat, who have web security products.
I've largely ignored them because from
199.19.248.0/21
they haven't been forwarding client addresses.
But recently I saw one from
198.135.124.0/23
where they do indeed forward client addresses.
The Via: header has in all cases been 'threatpulse'.
They don't appear to be getting blocked on my site.

[dstiles]

Bluecoat sometimes comes in with the UA...
Mozilla/4.0 (compatible;)
with a header...
HTTP_X_BLUECOAT_VIA

Or it did. Haven;t seen so many lately.

I think this combination is/was from bluecoat customers rather than from their own ranges. It seemed to be a sort of tester. I can't recall it ever actively addressing more than a single page at any given time. I blocked it with a warning but never had any indication this caused a problem.

Other than that, I have 6 blucoat IP ranges listed, one from AU the others from US, all of them blocked. I don't recall threatpulse at any stage but may have missed it.

[keyplyr]

Inside servermania.com:
104.144.0.0/16
104.144.0.0 - 104.144.255.255

Is this customer: yourprivateproxy.com:
104.144.7.0/24, 104.144.8.0/23
104.144.7.0 - 104.144.9.255

[dstiles]

Thanks for the B2Net range but I'll passon the proxies, thank you. There is too much leeway for seo merchants. :(

[keyplyr]

I didn't let them in either :)