Go-http-client/1.1

[TorontoBoy]

There's an old related thread titled Go 1.1 package http [webmasterworld.com] but it is closed. Don't know if a merge or addition is required.

This UA is now being used by 163data.com.cn, a Chinese bot(s) net that I have been tracking for a couple of years. They are not only prolific but pretty crafty. Usually their UAs are very anonymous, and therefore hard to pin down. I am surprised that they are so bold as to use an out-of-the-box package.

163data.com.cn has been implicated with the subverting and therefore banning of the Tor network within China [kau.diva-portal.org], pp10, so it is either brilliant or has connections to the Chinese government. I think the latter, hence my reason to track them so closely.

121.234.30.27
163data.com.cn usually uses only single IP address over range of Chinanet Fujian, but I have tracked them using IPs from most of the larger Chinese provinces, including Guangdong and Beijing.

[keyplyr]

There are literally dozens (maybe hundreds) of HTTP Clients that can be run from the server. They are constantly used by anyone wanting to access via HTTP.

If blocking, best to do so by UA.

I've always had difficulty getting exact company ranges from inside China's system. Lookups usually just list ISPs.

[dstiles]

Go-http-client/1.1 - I've been running a lot of web site security checks recently and found this client in my logs with a google IP (not bot IP). It appeared thanks to observatory dot mozilla dot org. I tried to whitelist it just for that service but gave up trying as it hopped around the cloud. I have no idea what it was trying to do but there were four hits per test.

[TorontoBoy]

I use observatory dot mozilla dot org to test web sites for general web security. I added a content security policy to my WP site, but this was not recognized by mozilla. It turns out there's a bug where sometimes even if you have a CSP it won't recognize it. Suffice to say that Observatory is a "work in progress".

It is good to know that they use Go-http-client/1.1. I usually ignore weird stuff from my IP in my log.

[keyplyr]

Suffice to say that Observatory is a "work in progress".

I block Go-http-client/1.1 but the Observatory gives my site a report anyway. The page I respond with is from my server, but a different config (different account) so all the UA needs is just the response header.

[keyplyr]

Looks like the Moz Observatory is still coming from different nodes of Google Cloud using Go-http-client/1.1.

One of the 3rd party reports the Observatory uses also uses the UA: Go-http-client/1.1 is securityheaders.io and comes from Digital Ocean ranges.

They seem to offer a nice set of header checks, with a few new fields that Moz doesn't include.

@TorontoBoy, after a little tweaking, I got the CSP to show in my headers and Moz recognized it giving me an A... however it blocked some of my JS and Adsense even though I allowed for them. I think it's because I redirect... anyway, I'll figure it out.

[LuckyLiz]

So, for those of us who aren't programmers and don't understand a lot of the acronyms and terms used here, should we just tell our hosting company to block anything that comes from Go-http-client/1.1 ?

[lucy24]

should we just tell our hosting company to block

Absolutely not. Why should they? On your individual sites, via htaccess-or-equivalent, you can block whatever you personally think should be blocked. Another site on the same server may choose not to block them. Both are equally valid choices.

If it is not in your power to set up your own access controls without your host's cooperation, change hosts.

[LuckyLiz]

@Lucy24 I guess I wasn't specific enough. Sorry! We have our own server. We outsource programming and server management and don't have anyone on staff who knows how to set up the blocking or knows anything other than basic html.

[keyplyr]

We have an ASP.net forum: [webmasterworld.com...]

Although setting up blocking filters isn't discussed often, that's where you would ask your questions regarding code to do so.

[TorontoBoy]

If you have different sites on your server, they might have different bot requirements and therefore ban rules. Otherwise your outsourced webmaster company should know how to do the ban.