Secure and other valid proxy sources

An escape from servers

[dstiles]

As may be obvious from some of my recent postings, I've been trying to improve the traffic for my clients by finding more "proxy" IPs. It began when one client complained of losing orders, although my actual opinion on that was summed up mostly, but not entirely, by "seasonal".

Over the past week-ish I've unblocked scansafe, ironport, websense, blackberry, opera, webroot, silk and flipboard within Amazon and similar: services I've blocked either because they hide within server farm ranges or their activity appeared in the past to be bot-like (there are more but I forgot to make notes!). There are also ones I already had cleared such as synetrix and research machines for UK education.

I've just discovered that zscaler is not only a secure proxy but has hundreds of small IP ranges. The only one I've found so far is within an internap range: 77.242.202.224/27. I say the only one: I recall the name going some way back but it was always a sub-range of a more popular server farm, which I then (ignorantly) blocked.

1. Does anyone have a reasonable list of zscaler proxy IPs?

2. I have evidence of at least one proxy IP within an otherwise-blocked MS range. Does anyone have a list? The sample one did not show up in DNS with any reasonable rDNS name so running a lookup on all IPs is probably not useful.

3. What other proxy ranges should I be looking at?

4. Are there similar in-server ranges that send valid traffic (eg Nokia within Amazon).

If there is any interest in this I will attempt to disentagle what I already have, assigning numbers to services.

[trintragula]

New zscaler?
8.28.150.0/24

Are you collecting Onavo? I understand keyplyr's misgivings, but still think they're a possible candidate.
203.190.121.0/24
which is also internap.

I've also seen a proxy at mileweb - a web accelerator which is part of Chinanet US.

I think there are a lot of web accelerator companies actualy. Wikipedia lists a few, though I've only been visited by a few of those.

For the last couple of weeks I've been collecting proxies in a DB table. I haven't been through them all yet, as I'm waiting for more to accumulate - which makes it easier to spot the proxy farms as distinct from infected machines. So far I have about 600, roughly a quarter of which forward a client address. I haven't tested any of them for public access.
I can summarize my findings at some point if there's any interest...

[dstiles]

Thanks for zscaler. New to me, too.

203.190.121.0/24 looks odd. In the middle of an APNIC range, hosted by internap with a user in UK with a gambling email address. Think I'll give that a miss, although I have set internap in general for onavo.

Chinanet thanks but I don't allow those.

I see a lot of proxies. If they are well-behaved in themselves and their users they get access. If not, rejected. Many of the proxies are simple firewalls for a home/business network.

[trintragula]

More zscaler this morning:
72.37.140.0/24
Italy this time.

Just for clarity - I've only seen a single proxy in each of these zscaler /24s.
I'm also assuming you're only interested in zscaler ranges that have been shown to have proxies - as the ASN for the above ranges has many more ranges that have not been listed here.

I'm largely ignoring proxies that don't forward a public IP (which is most of them).
Some of the proxies in academia seem to be getting abused by spammers. Presumably they don't even know...

I've just started recording which proxies are forwarding traffic that gets blocked. So far it seems to be mostly those proxies in China and AWS (though Silk seems to be getting though).

[dstiles]

> More zscaler this morning

Thanks. I also have three scansafe ranges in there:
72.37.171.0 - 72.37.171.255
72.37.244.0 - 72.37.244.255
72.37.248.0 - 72.37.249.255

> only seen a single proxy in each of these zscaler /24s.

Any number of them helps to identify them. Once identified I can sit back and ignore them - well, I'd like to but there are always servers that try using them. :(

> I'm also assuming you're only interested in zscaler ranges that have been shown to have proxies
> - as the ASN for the above ranges has many more ranges that have not been listed here.

True. I did go through those but the list did not include all proxy ranges.

> I'm largely ignoring proxies that don't forward a public IP (which is most of them).

A lot more desktops now run a proxy or proxying firewall - the latter often badly set up. :(

[trintragula]

I've seen several proxies from:
80.187.0.0/16
This is T-mobile in Germany, and I think it's probably the web-accelerator they run.
Today's visit is not something I'd want to block as it shows several signs of being a plausible visit - not least a plausible client IP in the XFF header. It's a shame so many proxies fail to send a Via header.

I also had a visit from the Nokia Amazon range yesterday. The proxy Via header contained the keyword Novarra, which is a company Nokia bought to do web acceleration.

It looks like a lot of companies are springing up to do web acceleration for mobiles (which often have slow and expensive 'data services' (web access)). There's a short list of them on wikipedia, and I have seen visits from a few of them in the last couple of weeks. Looks like Kindle/Silk is the same idea, but cloud-based.
I suspect that while some are third party, some may be built in to the phone service and may not easily be bypassed except by someone fairly knowledgeable. There seem to be quite a few Q&As on the web about how to do that for specific services. So insisting that people don't use them may be unreasonable in some cases.

Yahoo! Korea
114.141.40.0/21
The specific IP was: 114.141.47.173
though the client was my pt-PT friend, so that was blocked.... Not sure I'd want to block the proxy though.

[trintragula]

Mileweb seems to be part of the cloud doing data compression for ucweb's UC browser.
The browser is Chinese made but I think I saw a human from the UK using it today.
C3 networks (which has IPs in China and the US) apparently also support the ucweb cloud.

Some more scansafe:
184.150.236.42/31
184.150.236.50/31
184.150.236.58/31
184.150.236.66/31

I wish there were a way to free-text-search the whois records ("show me any record containing the string 'scansafe'"). I saw two from those ranges, the rest I found by guesswork. :(

[trintragula]

More Onavo
212.118.227.58
contacts are at internap - no gambling this time. :)
(EDIT: if you've got the internap ranges open for Onavo you've presumably got this open anyway).

Former bogon (less than a month ago),
45.33.130.173
now Hurricane electric, hosting CloudMosa.
Not obviously promising, but:

CloudMosa — the maker of Puffin, a cloud-powered mobile browser — today revealed it has secured at least $18 million in funding and may raise $4.9 million more as it charges past the 25 million user mark.

_{-- Venturebeat Dec, 2014}

Looking at who was coming through the proxy:
Mozilla/5.0 (X11; U; Linux x86_64; it-it) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.114 Safari/537.36 Puffin/4.1.4.1387AT

Clearly identified as Puffin, which is nice.
It has it-it in the UA string, it-it in the language header. From FastWeb. Italy.
I think my visitor might be Italian...

[dstiles]

Thanks for the extra ranges! :)

Thanks for Puffin. Quite impressed by their FAQ page with the IP and UA info. :)

[trintragula]

Sure thing.

I had a few proxy visits recently from exa networks.
82.219.0.0/16

Via header contains 'ExaProxy'

This proxy apparently supports their SurfProtect product which seems to be aimed at schools.
Content filtering (i.e. keeping kids away from unsuitable websites - I hope they decided mine is okay).

The source code for the proxy is up on github, so I wouldn't assume all ExaProxy hits are from Exa networks.

The up-to-date Chrome/41 visit was followed up by a couple of Firefox/10 visits, which I think may have solved a mystery...

Not surprisingly, this visit was not blocked by my pet monster.

[trintragula]

Re: puffin:
I hadn't spotted the list of ranges in their FAQ page (or the FAQ page for that matter). Interesting!
Given the new Hurricane Electric range I guess their list is out of date now.
The last range in their list is from XO Communications.

They've updated the Puffin version, but the chrome version they send appears to be frozen in time...

[keyplyr]

If anyone would be interested to start a thread about proxies...

Aren't you doing just that?

[keyplyr]

Are you collecting Onavo? I understand keyplyr's misgivings, but still think they're a possible candidate.

What "misgivings" are you attributing to me regarding Onavo proxies? I allow most all proxies except those that have shown to be constantly abused.

[trintragula]

If anyone would be interested to start a thread about proxies...

Aren't you doing just that?

Well, no. I'm just following up dstiles original fairly specific questions with more examples from the categories he's interested in. I hope. It's another open-ended enumeration, and one I'm happy to participate in for a while because it helps me to understand the issues.
Though I have a suspicion he might be happier if I just shut up and posted the CIDRs...

My suggestion was to start a wider discussion, which would help me, and perhaps other people.
I could probably actually contribute quite a lot to that, but I've spent a lot of my time on here posting long topics about other forms of robot blocking that no-one seems to be interested in, so I thought this time I would ask if someone else wanted to. The three week silence seems to have answered that one.
The least discouraging reason I can think of for that is that everyone here is too busy.

Regarding Onavo - I think the last thing you said about them was that they were staying blocked on your site based on what you'd read about them - which is fair enough. Our needs are different.

[keyplyr]

...this time I would ask if someone else wanted to. The three week silence seems to have answered that one.

Not a lot of members check request headers, which is kinda required to efficiently detect/track proxies beyond just doing internet searches for lists.

I admit that since I travel so much, I don't keep at it the way I should either. When I'm home on a desktop with processing power I'm diligent with checks, but on the road with a phablet it's futile. When I get home from a long gig, I usually have a ton of data piled up and a lot gets dumped.

I have benefited from the proxy discussion so far and would like to see it continue (in whatever form) even if I don't contribute much.

RE: Onavo - It isn't the proxy ranges I was against, it was just the file caching I was trying to stop; having given up on that idea since it seemed to be more wide-spread than I thought, especially with mobile, making it the proverbial lost cause :)

[bobothecat2]

Another Zscaler:

199.168.148.0/22

[keyplyr]

This thread has drawn my attention to making sure I'm not inadvertently blocking friendly visitors who may be using a proxy. Checking my filters, I discovered that despite my knowledge to the contrary, I was blocking "UNTRUSTED" in UA string.

EXAMPLE: NokiaX2-05/2.0 (08.35) Profile/MIDP-2.1 Configuration/CLDC-1.1 UCWEB/2.0 (Java; U; MIDP-2.0; en-US; NokiaX2-05) U2/1.0.0 UCBrowser/9.5.0.449 U2/1.0.0 Mobile UNTRUSTED/1.0

This visitor came in from proxy at:
Chinanetcenter.com
8.37.224.0/20
8.37.224.0 - 8.37.239.255

The mobile phone's config may add "UNTRUSTED" to the UA string if outside the normal mobile network and using a network that does not give the proper credentials, like an unregistered proxy or an improperly signed cert, I guess I added the block at some point in the past without thinking, most likely a knee-jerk reaction to something..

Doing a few checks, this UA attribute is wide-spread and if blocked, may be affecting friendly traffic. Just a FYI in case anyone else was/is doing the same.

[trintragula]

Digitally signed certificates seem to show up in a number of contexts, and their absence generally indicates that an additional but optional layer of security is not being employed. Whether that's a problem or not depends on the context, I guess.

I've had a proxy visit from another zscaler range:
104.129.196.0/24
or
104.129.196.0/23
or
104.129.192.0/20
depending on where you ask and how wide you want to cast the net.

[trintragula]

I'm getting a steady flow of UCBrowser mobile traffic, proxied through Chinese-owned proxies. None of this is getting blocked by my traps. It appears to be humans on e.g. Verizon in the US, or Virgin media in the UK, but also BRIC countries.
UCBrowser has 500 million users in the BRIC block and 1% of mobile browsers in the US (comparable with Opera or Blackberry).
Because of this, and leaving aside Baidu, I'm currently allowing through more Chinese traffic than I'm blocking.

@keyplyr
Your chinanetcenter visitor IP is also listed as Mileweb, which is where I've been seeing the bulk of my UCBrowser visitors.

[keyplyr]

Not a proxy per se, TunnelBear makes several apps which connects the social media user to their VPN which comes from Digital Ocean:

192.241.128.0/17
192.241.128.0 - 192.241.255.255

Started noticing Facebook referrals from this range. Not a lot (under a dozen per day) but IMO enough to poke a hole in DigitalOcean for FB users (referral & UA for iPhone/Android app users):

Note: If you don't court social media, then this is probably not worth the effort.

[keyplyr]

UA: GoogleFriendConnect/1.0
Host: google-proxy-64-233-172-143.google.com

64.233.160.0/19
64.233.160.0 - 64.233.191.255

I'm currently allowing through more Chinese traffic than I'm blocking.

As an aside: since I launched as mobile responsive 8 months ago, I rethought my Asia blocking. Up until this time, Asia seemed to be mostly a PITA, so just blocking Korea, China, Taiwan, Vietnam, Laos, et al seemed the solution. However, since going mobile and realizing that they're a HUGE mobile traffic source, I removed the blocks & started courting that demographic, mostly through social media and allowing the plethora of apps. They don't buy many of my products or services, but daily traffic & Adsense revenue has increased significantly.

[dstiles]

keyplr - do you have UAs for tunnelbear accesses, please?

I have 64.233.172.0/23 open as proxies.

[keyplyr]

AFAIK tunnelbear did not add any specific UA attribute, at least the hits I saw. Tunnelbear makes a couple apps which may have a specific UA string, but the Facebook referrer followed a link to my site dragging a browser UA. I believe this one was just using their pipeline (VPN.) Didn't check header as I was on my phablet sitting in a bar drinking Irish beer :)

[keyplyr]

RE: Google proxy 64.233.160.0/19, 64.233.160.0 - 64.233.191.255

It just occurred to me this range is also used for image retrieval when someone posts a link to your site at Google+. Similar to Facebook, this is a good thing IMO, making your link attractive to visitors.

FB or Google+ will usually just grab the first large image on the page, however you can designate the image used by placing this tag in <HEAD> section of your page mark-up:

<link rel="image_src" href="hhttp://www.example.com/file.jpg">
(HTML backward compatible)

- or -

<meta property="og:image" content="http://www.example.com/file.jpg">
(HTML5 compatible)

Note: image must be at least 200x200px.

This tool: [developers.facebook.com...] will test the displayed image.

[dstiles]

keyplr - thanks for tunnelbear (non)-info. :)

Does anyone have opinions re: privax? I block it at present but it claims to provide hundreds of free proxies, which I am not completely happy about. I've tried half a dozen pages of their web site from ixquick's results but none of them seem valid: all go to hidemyass site which also does not seem to have valid pages.

[keyplyr]

Privax (now owned by AVG) is HideMyAss

IMO the value in allowing proxy access is determined by the intended or real use of the proxy - so HideMyAss has never seemed like a logical hole to punch (no pun intended.) Here are my IP lists for them:

#HMA 95.154.245.64 - 95.154.245.127
#HMA 212.38.167.192 - 212.38.167.255
#HMA 212.38.170.64 - 212.38.170.127
#HMA 212.38.170.128 - 212.38.170.191
#HMA 212.38.170.0 - 212.38.170.63
#HMA 78.129.168.0 - 78.129.168.127
#HMA 78.129.150.128 - 78.129.150.255
#HMA 78.129.168.128 - 78.129.168.255
#HMA 78.129.150.0 - 78.129.150.127
#HMA 87.117.232.128 - 87.117.232.255
#HMA 109.169.17.0 - 109.169.17.127
#HMA 78.129.160.0 - 78.129.160.127
#HMA 78.129.171.0 - 78.129.171.127
#HMA 78.129.160.128 - 78.129.160.255
#HMA 109.169.4.128 - 109.169.4.255
#HMA 82.145.62.128 - 82.145.62.255
#HMA 62.233.42.0 - 62.233.42.127
#HMA 62.233.42.128 - 62.233.42.255
#HMA 62.233.37.0 - 62.233.37.127
#HMA 62.233.34.128 - 62.233.34.255
#HMA 62.233.41.128 - 62.233.41.255
#HMA 62.233.41.0 - 62.233.41.127
#HMA 62.233.34.0 - 62.233.34.127
#HMA 185.25.84.0 - 185.25.84.127
#HMA 5.102.170.0 - 5.102.171.255
#HMA 91.238.214.0 - 91.238.215.255
#HMA 87.117.238.64 - 87.117.238.127
#HMA 185.25.84.128 - 185.25.84.255
#HMA 5.56.23.128 - 5.56.23.255
#HMA 46.229.224.128 - 46.229.224.255
#HMA 130.185.105.128 - 130.185.105.255
#HMA 31.7.187.0 - 31.7.187.255
#HMA 185.25.86.0 - 185.25.86.255
#HMA 185.25.87.0 - 185.25.87.255
#HMA 185.25.85.0 - 185.25.85.255
#HMA 178.16.27.0 - 178.16.27.255
#HMA 185.25.84.0 - 185.25.87.255
#HMA 37.46.115.0 - 37.46.115.255
#HMA 178.73.212.96 - 178.73.212.127
#HMA 178.73.198.0 - 178.73.198.63
#HMA 46.246.89.0 - 46.246.89.127
#HMA 188.126.73.128 - 188.126.73.255
#HMA 46.246.123.0 - 46.246.123.255
#HMA 162.211.178.0 - 162.211.178.255
#HMA 162.211.179.0 - 162.211.179.255
#HMA 66.187.64.0 - 66.187.64.255
#HMA 66.187.68.0 - 66.187.68.255

Note: I have never blocked HMA categorically by range, but have occasionally seen trouble from one or two users. Those that don't bring trouble, I don't notice :)

[dstiles]

Thanks for the info, keyplr. All entered. Now to make decisions. :)

[aristotle]

This IP showed up in Statcounter, but not in my logs.

Statcounter entry:
Google (209.85.238.71) 0 returning visits
Miami, Florida, United States
(No referring link)
8 May 22:27:03

IP Lookup:
IP: 209.85.238.71
Hostname: rate-limited-proxy-209-85-238-71.google.com
ISP: Google
Organization: Google
Services: Suspected proxy server
Type: Corporate
Assignment: Static IP
State/Region: California
City: Mountain View

I can't find an entry in my logs for this IP or time. Maybe someone can explain it, because I don't understand it.

Edit P.S. Now it has occurred to me that this could have come from a saved page on someone's disk that triggered the Statcounter code. But I still don't understand the proxy role.

[dstiles]

My note on that /24 is: "rate-limited-proxy - adsbot etc" which suggests I once got annoyed at its bot and that it MAY be a proxy, in which case tough 'cause it's blocked due to G's terrible DNS practices.

[lucy24]

#HMA 185.25.84.0 - 185.25.84.127
#HMA 185.25.84.128 - 185.25.84.255
#HMA 185.25.86.0 - 185.25.86.255
#HMA 185.25.87.0 - 185.25.87.255
#HMA 185.25.85.0 - 185.25.85.255

or, in short,

#HMA 185.25.84.0 - 185.25.87.255

(Dunno about anyone else, but I get annoyed when I hear about anything even remotely suspect coming from 185. Oi! You know there are real people desperate for an IP block over there.)

Feeding all those HMAs into my records, I find that a fair number of them are sublets from Iomart or, less often, Portlane. Any relationship, or just all-around coincidence?

[keyplyr]

Sublets is why they are broken into small ranges. Maybe not a good idea to combine them.