Secure and other valid proxy sources

An escape from servers

[dstiles]

As may be obvious from some of my recent postings, I've been trying to improve the traffic for my clients by finding more "proxy" IPs. It began when one client complained of losing orders, although my actual opinion on that was summed up mostly, but not entirely, by "seasonal".

Over the past week-ish I've unblocked scansafe, ironport, websense, blackberry, opera, webroot, silk and flipboard within Amazon and similar: services I've blocked either because they hide within server farm ranges or their activity appeared in the past to be bot-like (there are more but I forgot to make notes!). There are also ones I already had cleared such as synetrix and research machines for UK education.

I've just discovered that zscaler is not only a secure proxy but has hundreds of small IP ranges. The only one I've found so far is within an internap range: 77.242.202.224/27. I say the only one: I recall the name going some way back but it was always a sub-range of a more popular server farm, which I then (ignorantly) blocked.

1. Does anyone have a reasonable list of zscaler proxy IPs?

2. I have evidence of at least one proxy IP within an otherwise-blocked MS range. Does anyone have a list? The sample one did not show up in DNS with any reasonable rDNS name so running a lookup on all IPs is probably not useful.

3. What other proxy ranges should I be looking at?

4. Are there similar in-server ranges that send valid traffic (eg Nokia within Amazon).

If there is any interest in this I will attempt to disentagle what I already have, assigning numbers to services.

[keyplyr]

4. Are there similar in-server ranges that send valid traffic (eg Nokia within Amazon).

I think you have these, but here they are again...

Inside Amazon's 54.208.0.0/13 are Nokia Express:

54.209.248.0/22
54.209.248.0 - 54.209.251.255

54.236.252.0/22
54.236.252.0 - 54.236.255.255

54.244.56.0/21
54.244.56.0 - 54.244.63.255

54.246.252.0/22
54.246.252.0 - 54.246.255.255

[lucy24]

Over the past week-ish I've unblocked scansafe, ironport, websense, blackberry, opera, webroot, silk and flipboard within Amazon and similar:

Long ago I was obliged to unblock the websense range at
208.87.232.0/21
because it's used as a proxy by my local county government. (In reality it's always the identical IP, aa.bb.cc.dd, but I just unblocked the whole thing.)

Keep an eye out for any requests for the favicon and apple-touch-icon-- especially if your html doesn't link to them by name-- accompanying blocked page requests. Requests are almost bound to be human, barring a handful of easily identified robots (Google, DuckDuckGo).

[dstiles]

keyplr - thanks. I didn't have the final range.

Lucy - thanks for the tip re: favicon etc. I keep favicon in an images folder, linked to within the header. It's a left-over from the invasive icon bots and fetchers of yore. Apple icons - I get loads of requests for those but all are returned 404.

[keyplyr]

Apple icons - I get loads of requests for those but all are returned 404.

I used to just let these requests return a 404, but then discovered that Apple mobile devices used these various sized png files as home screen icons when they save a link to your page. If the file is not found, a default image is used. Depending on the device, there are a couple different default icons, neither of which is very attractive IMO. I would rather control what this icon looks like that is representing my web page.

So I made a nice looking square, branded png image in the different sizes & names that are requested:
apple-touch-icon-57x57.png
apple-touch-icon-57x57-precomposed.png
apple-touch-icon-72x72.png
apple-touch-icon-72x72-precomposed.png
apple-touch-icon-76x76.png
apple-touch-icon-76x76-precomposed.png
apple-touch-icon-114x114.png
apple-touch-icon-114x114-precomposed.png
apple-touch-icon-120x120.png
apple-touch-icon-120x120-precomposed.png
apple-touch-icon-144x144.png
apple-touch-icon-144x144-precomposed.png
apple-touch-icon-152x152.png
apple-touch-icon-152x152-precomposed.png

The newer tablets & phone seem to just request these:
small.png (70x70)
medium.png (150x150)
large.png (310x310)

And some legacy mobile devices request these:
apple-touch-icon.png (57x57)
apple-touch-icon-precomposed.png (72x72)

[lucy24]

Apple icons - I get loads of requests for those but all are returned 404.

Oh, now that's interesting. So they ask for
example.com/favicon.ico
even if your 403 page explicitly says where to find the favicon? (Or whatever page the Firefox 6 UA gets on your site. Mine gets a redirect to the Old Browsers page. The former null UA got a flat 403.)

[dstiles]

Further to the Nokia/Amazon association, does anyone know if there is a similar association between Nokia and MS cloud (azure)? I'm seeing an occasional proxy access using 137.135.176.nn that looks valid but is being rejected at present due to the cloud bit. The IP may vary slightly - I haven't been paying attention beyond the 137.135 bit.

keyplr - thanks for the png collection. :) It will be a while before I ever get around to it, though.

Lucy - they do not get a favicon with any form of reject: it's a very simple header for those pages. I only include a favicon path in real page headers. I have to say Firefox's method of guessing the favicon location annoys me. If it waited until the header was loaded it could be more certain of getting it. It does load eventually, of course.

[lucy24]

The newer tablets & phone seem to just request these:

If you can't be bothered to create twenty different apple-touch-icons -- which is understandable, since they change their defaults every other week -- they will eventually ask for
apple-touch-icon-precomposed.png
and will go away contented. You just have to decide what (actual) size to make it. Currently 120x120 seems to be the most popular. It's exactly equivalent to a favicon; for example it may show up in your browser history or favorites, not just on the desktop.

[dstiles]

Further to zscaler...

I found a number of (I think!) proxy IPs at...
[robtex.net...]

My current zscaler list is below, although I'm pretty sure I will need to add more and possibly prune some I have. The robtex list, for example, does not include a French one I found a couple of days ago.

42.99.164.0 - 42.99.164.255 (Singapore)
64.215.22.0 - 64.215.22.255 (Brazil)
77.242.202.224 - 77.242.202.255 (French)
95.172.74.0 - 95.172.74.255 (UK)

104.129.192.0 - 104.129.207.255 (US) (this is a zscaler cloud containing...)
104.129.192.0 - 104.129.192.255
104.129.194.0 - 104.129.194.255

175.45.116.0 - 175.45.116.255 (Australia)
211.13.192.0 - 211.13.223.255 (Japan)

[dstiles]

Touch icons - Thanks, Lucy. On my todo list. :)

[keyplyr]

they will eventually ask for apple-touch-icon-precomposed.png and will go away contented.

I have never seen this. From my experience the Apple device requests a certain file, say for example: apple-touch-icon-152x152-precomposed.png. If the file is not found, the server responds with a 404 error. I have never seen the same IP then switch to requesting an alternate file named: apple-touch-icon-precomposed.png. They request what they request. If they don't get it, they use the default icon supplied from the devices icon library.

What I do see however, are these specific files requested less and less as time goes on and Apple users replace their old handsets/tablets for newer models or update OS versions. Looking through the last 6 months of logs, I now mostly see hits for: small.png, medium.png & large.png (sizes supplied in earlier post) but there remains a significant number of requests for the older specific type/style of Apple icons as well.

Note: this observation from a site with 10k to 13k daily page loads, where Safari persists as top mobile browser (although since other browsers also falsely identify as Safari, this stat is far from 100% accurate.)

[lucy24]

I have never seen this.

I searched raw logs for the string "apple-touch-icon". Sometimes they make just one request. But when there's a series, the pattern is
apple-touch-icon-more-stuff-here 404
apple-touch-icon-other-stuff-here 404
and after several guesses they ask for either
apple-touch-icon-precomposed.png
or
apple-touch-icon.png
(both of which I've got on the site whose logs I checked).

The bare apple-touch-icon has a preferred default size, which has changed over the years. Worse, the handling of "precomposed" has also changed over the years. It has to do with rounding edges and optionally also the highlighted bit in the middle (supposed to make the whole thing look shiny or reflective).

Another thing that has changed over the years is the increasing difficulty of finding information about the apple-touch-icon. Currently they list four sizes [developer.apple.com].

Oh, and the mobile user makes a fresh apple-touch-icon request with every page visit, unlike the favicon which is only requested once. That's why logs seem so bloated.

[keyplyr]

Well, I have no experience seeing what you see, but thanks for the developer.apple.com link. I just added 4 180x180 icons (tagged with dimensions, and not tagged.)

BTW - I use rounded corner tiles for them all. Not sure about the "shiny" middle. Guess it depends on how the light hits it :)

Another observation. I have seen my icon on the home screen of several size iPhones. It's my suspicion that the iOS sizes the icon to fit the Home screen or app tray. It would prefer the correct size, but if not found I believe it will enlarge/reduce size to fit.

This belief comes from the fact that until 5 minutes ago I did not have a 180x180 icon (compressed or not) but whatever icon of mine the device is using, it displays relatively the same size as the others on the iPhone6 Plus, which I suspect is the one wanting the 180x180 version.

[dstiles]

Looking at my new posting re: com.apple.webkit, I notice ONLY 301 and 302 responses for touch icons. No 404's.

It is possible that this response is pre-programmed into later versions of IIS.

[keyplyr]

It is possible that this response is pre-programmed into later versions of IIS.

I would not think that, no.

What I would think is the requests are coming in for either a www and you don't use www, or the other way around. Then the server is giving them the redirect to the 404, but only the 301 or 302 is showing in the log. My server does the same thing.

[lucy24]

It's my suspicion that the iOS sizes the icon to fit the Home screen or app tray. It would prefer the correct size, but if not found I believe it will enlarge/reduce size to fit.

Yes, that's one of the things that used to be explained pretty intelligibly in apple docs but now I can't find it.

They like full-color png with an alpha channel, so you can really cut loose. Not like old-fashioned icons that used 16 or 256 colors. (I've still got a few of those lying around in ancient graphics files.)

[trintragula]

I'm considering blocking the tuple of the proxy and the address it claims to be forwarding for, thus treating the proxy and its client as a longer IP address.

[dstiles]

On the original topic of this thread, I'm still enabling genuine proxies as I come across them in logs. I've also noticed some new proxy-related headers, which I must investigate ASAP. In fact, the whole issue of headers is changing, with some of those "must be present" ones now being (may not be present". I think some of the blame may be on SSL services, possibly via proxies, but I do not think that's all it is. It looks as if certain browsers (G springs to mind) are dropping such headers.

Apropos enabling proxies, I found two MS proxy IPs today (actually been seeing one of them for a while now - see earlier posting above) and have enabled 137.135.176.0/24. This may be temporary: the IPs, tagged "london" in the Via field, do not have a helpful rDNS. The specific IP that began the search for numbers ended in 71, if it helps.

I have also added a few extra zscaler ranges to my list. I will try to remember to add them here in due course.

[dstiles]

Sorry for the delay, folks: variety of reasons.

The following lists are almost certainly not complete; they are just what I've discovered. Nor do they include the many UK schools proxies I have opened.

zscaler...
31.186.238.0 - 31.186.238.255
42.99.164.0 - 42.99.164.255
64.215.22.0 - 64.215.22.255
77.242.202.224 - 77.242.202.255
82.220.1.192 - 82.220.1.223
95.172.74.0 - 95.172.74.255
104.129.192.0 - 104.129.192.255
104.129.194.0 - 104.129.194.255
175.45.116.0 - 175.45.116.255
211.13.192.0 - 211.13.223.255

websense...
85.115.32.0 - 85.115.63.255
86.111.216.0 - 86.111.223.255
91.194.158.0 - 91.194.159.255
204.15.64.0 - 204.15.71.255
208.80.192.0 - 208.80.199.255
208.87.232.0 - 208.87.239.255

scansafe...
46.255.40.0 - 46.255.47.255
69.10.152.0 - 69.10.152.255
69.174.58.0 - 69.174.58.255
69.174.87.35 - 69.174.87.36
69.174.113.0 - 69.174.113.255
70.39.176.0 - 70.39.177.255
70.39.231.0 - 70.39.231.255
72.37.171.0 - 72.37.171.255
72.37.244.0 - 72.37.244.255
72.37.248.0 - 72.37.249.255
80.254.144.0 - 80.254.159.255
202.167.250.0 - 202.167.250.255

ironport...
108.171.128.0 - 108.171.159.255
204.15.80.0 - 204.15.83.255
208.90.56.0 - 208.90.63.255

opera...
37.228.104.0 - 37.228.111.255
64.255.180.0 - 64.255.180.255
80.239.242.0 - 80.239.243.255
82.145.208.0 - 82.145.223.255
91.203.96.0 - 91.203.99.255
107.167.96.0 - 107.167.127.255
141.0.8.0 - 141.0.15.255
185.26.180.0 - 185.26.183.255
195.189.142.0 - 195.189.143.255
213.236.208.0 - 213.236.208.255

google (be sure to block the many baddies (eg google) who run bots through these!)...
64.233.172.0 - 64.233.173.255
66.102.6.0 - 66.102.9.255
66.249.80.1 - 66.249.85.255
66.249.88.1 - 66.249.93.254

barracuda...
64.235.144.0 - 64.235.159.255

yahoo...
66.163.171.44 - 66.163.171.45
68.142.243.48 - 68.142.243.81
68.142.243.93 - 68.142.243.99
68.180.211.0 - 68.180.211.255
69.147.77.24 - 69.147.77.30
69.147.114.45 - 69.147.114.45
69.147.114.64 - 69.147.114.66
69.147.114.104 - 69.147.114.109
69.147.115.58 - 69.147.115.61
69.147.115.64 - 69.147.115.75
69.147.115.76 - 69.147.115.79
87.248.122.64 - 87.248.122.95
98.139.240.154 - 98.139.240.159
98.139.241.56 - 98.139.241.85
98.139.241.224 - 98.139.241.252
98.139.242.196 - 98.139.242.254
98.139.243.1 - 98.139.243.66
206.190.36.64 - 206.190.36.122
206.190.37.0 - 206.190.37.23
209.131.38.42 - 209.131.38.43
209.131.38.70 - 209.131.38.71
209.131.38.88 - 209.131.38.92
209.131.38.128 - 209.131.38.139
209.131.39.48 - 209.131.39.79
209.131.39.60 - 209.131.39.63

blackberry...
68.171.224.0 - 68.171.255.255
74.82.64.0 - 74.82.95.255
93.186.16.0 - 93.186.31.255
131.117.168.0 - 131.117.175.255
178.239.80.0 - 178.239.95.255
206.53.144.0 - 206.53.159.255

messagelabs...
85.158.136.0 - 85.158.143.255
95.131.104.0 - 95.131.110.255
103.9.96.0 - 103.9.99.255
117.120.16.0 - 117.120.23.255
193.109.254.0 - 193.109.255.255
194.106.220.0 - 194.106.221.255
216.82.240.0 - 216.82.255.255

ixquick - not really a public proxy but they have a proxy option to launch indexed web sites...
37.0.86.2 - 37.0.86.6
37.0.86.20 - 37.0.86.25
64.71.175.0 - 64.71.175.255
74.82.58.0 - 74.82.58.255
213.144.235.192 - 213.144.235.223
213.247.119.230 - 213.247.119.237
216.218.239.0 - 216.218.239.255

[Hoople]

Ironport and barracuda are hardended Unix appliances for email anti-SPAM and web trafic sanitization. They are almost exclusively on top of a firewall appliance positioned as a gateway between a corporation's desktop browsers and the internet.

IMHO there is a zero percent likelyhood of a publicly accessable port to access these to route traffic back to the internet. Just a FYI from someone who has deployed them in the past. <G>

[trintragula]

What categories of proxy are you interested in? What's your criterion? What does secure or valid mean?
Do they have to be public, or is your goal actually to avoid blocking real customers (and presumably block proxies used for other purposes where possible)?
I could probably dig up a list of a couple of hundred TOR exits, for example.
Right now I don't keep track of headers for more than a day, but I see a dozen or so proxies with identifiable client IPs in my log at the moment.
Aside from candidates on your list I see several from China (which in the past have had OVH client IPs at least once) and also something called doubleroute.jp, which according to vpngate.net is a public VPN relay. Is that of interest?

[wilderness]

FWIW
Cisco Systems Ironport Division
208.90.56.152 - - [27/Feb/2012:00:31:00 +0000] "GET / HTTP/1.1" 200 6193 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
208.90.56.152 - - [27/Feb/2012:00:31:01 +0000] "GET / HTTP/1.1" 301 234 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"
208.90.56.152 - - [27/Feb/2012:00:31:01 +0000] "GET / HTTP/1.1" 200 6193 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0.1) Gecko/20100101 Firefox/4.0.1"

[trintragula]

Incidentally: the google proxies are the major way I've seen IPv6 traffic arrive at my site; in fact I'm not sure I've seen it arrive any other way. Certainly not since I started paying attention.
Since IPv6 increased from a drip to a slow trickle in the last year I've recently taken the time to make my blocker IPv6-aware. Though it has yet to need to stop any IPv6 traffic - whether by proxy or not.

[lucy24]

Heh. At one time I blocked one particular websense range. (I think 208.80.something.) I had to remove the block because it turned out that my county government-- collectively-- routes all requests through an exact IP address at this proxy. So they're humans who had no choice in the matter. Luckily someone had the sense (haha) to hunt down an IT person who could figure out why everyone was getting the door slammed in their faces.

[dstiles]

Hoople - Ironport and barracuda - thanks. I do see them regularly, though, and I don't think its always the same source as I sometimes block the Forwarded-For IP for being on a server and other times not.

trintragula - generally public. Things like squid are common on (eg) linux machines with local IPs and get blocked or not depending on their behaviour. I'm not interested in TOR. Nor China. I've found a lot of so-called public proxies are anonymous and they generally get stopped on their users' behaviour. - Apropos ipv6 - my server does not approve of that <sniff> :)

wilderness - thanks but I noted those under ironport - cisco/ironport are a single entity (one took over the other, can't recall which).

[wilderness]

wilderness - thanks but I noted those under ironport - cisco/ironport are a single entity (one took over the other, can't recall which).

Ironport and barracuda are hardended Unix appliances for email anti-SPAM and web trafic sanitization. They are almost exclusively on top of a firewall appliance positioned as a gateway between a corporation's desktop browsers and the internet.

[trintragula]

Apropos ipv6 - my server does not approve of that <sniff>

Even so, IPv6 addresses can show up in the XFF header. So far I think I've only seen this from google's proxies. That v6 traffic comes from lots of places: comcast, road-runner, etc... largely US, but I did see some from Deutsche Telekom, I think - unlike the Americans they do actually have v6 rDNS. I seem to be blocking about half of proxied visitors, but so far the v6 traffic has been well-behaved (probably because there's so little of it).

Zenmate (hosted at Leaseweb) is a free proxy service. I'm pretty sure I've seen real people using this.

See also UCWeb UCBrowser - compare opera mini?

[dstiles]

wilderness - Ironport - typical entry in my logs...

Ironport IP: 108.171.128.nnn - part of ironport/cisco range
FwdFor IP: 149.126.110.nnn - UK IP in VTESSE range

This, to me, is "anyone who has signed up to ironport can use their proxies".

[dstiles]

I have just found a log entry that has resulted in expanding the scansafe range 69.174.87.35 - 69.174.87.36 to 69.174.87.0/24

[trintragula]

If anyone would be interested to start a thread about proxies generally and what to do about them from a blocking point of view, it's possible that I'm not the only person who would find that useful. If this is the right forum for that...