DownloaderChrome

[lucy24]

Does anyone know what the heck this is?

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36

It doesn’t appear to be a human addon. They’ve been coming around since the beginning of the month, always with that exact UA. I haven't noted the IPs more exactly, but they're all in 34, 52, and 54 so presumably AWS.

Curiouser still: Their only request, ever, is for robots.txt, always with a referer in one of these four forms (the site is http://example.com):
http:// example.com
http:// www.example.com
http://example.com/robots.txt
http://www.example.com/robots.txt
Yes, that's an actual space in the first two, not an artifact of my log-wrangling. I double-checked.

As you can imagine, googling this exact name is not a fruitful endeavor.

[keyplyr]

Looks like a downloader for Chrome :)

Actually, it's probably just a bot scraping data. A web search returns server logs with common mentions of this UA.

Please post the IP range when you report UA. That is relevant information to determine who/what this really is.

[fix typo]

[edited by: keyplyr at 7:26 pm (utc) on Apr 26, 2018]

[jmccormac]

Could be a bot but there's a report of Adware of that name in Bleepingcomputer.

Regards...jmcc

[TorontoBoy]

52.41.56.255[02/Apr/2018:12:25:15 GET /robots.txt HTTP/1.12001285 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.201.164.157 [05/Apr/2018:04:48:39 GET /robots.txt HTTP/1.12001285 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.217.117.174 [05/Apr/2018:19:50:02 GET /robots.txt HTTP/1.12001285 http://www.example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.211.123.247 [06/Apr/2018:01:05:15 GET /robots.txt HTTP/1.12001285 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.245.177.115 [11/Apr/2018:07:13:11 GET /robots.txt HTTP/1.12001414 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.187.172.150 [11/Apr/2018:22:32:19 GET /robots.txt HTTP/1.12001414 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.212.173.132 [12/Apr/2018:05:18:18 GET /robots.txt HTTP/1.12001414 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.212.173.132 [12/Apr/2018:05:18:18 GET /robots.txt HTTP/1.12001414 http://www.example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.202.187.111 [24/Apr/2018:05:24:28 GET /robots.txt HTTP/1.12001478 http:// www.example.com Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36

All are Amazon AWS IP, which I ban, with the exception of robots and my 403. I could not find the origin in a Google or Baidu search. There is nothing on Git. Many other sites have also seen this UA and have reported so. All my hits have been with different IPs from AWS, and with one exception, never repeated. They don't look at any other resource other than robots.txt.

Curious. This is a home grown bot of unknown origin and not a copycat.

[lucy24]

never repeated

Do you mean from one day to another, or within the same day? Mine tend to come in clumps of 2 or 3 from the same IP in rapid succession, and when this is the case they toggle the pseudo-refererer, adding or removing the www. Technically they are probably changing the host itself, and setting the referer to match, but I don't log headers on 301 responses--mainly because I don't know how to--so I can't be sure.

But yes, now that you mention it, it's a new IP every time, or every clump.

If they were requesting anything other than robots.txt they would be blocked due to one absent header.

[TorontoBoy]

I posted all my sightings from my 2018 April log. If you wish I could post my March sightings.

They clump only in 2s, the same IP per day, a cycle (clump!) every 3-4 days, but 2 different UAs. Only one UA is DownloaderChrome. The other UA is: "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0". Either could be first, but usually it is DownloaderChrome. Only once was DownloaderChrome used 2 times and the other not used at all. IPs are rarely repeated between cycles for the month of April.

The first referrer is always http://example.com/robots.txt. The second referrer is always http://www.example.com/robots.txt. They seek nothing else, and if they did I serve them 403s.

[keyplyr]

All are Amazon AWS IP

This is why we need the IP rangs(s) in the 1st post. That pretty much confirms it's not human.

Just to make sure, I blocked the UA, then downloaded a couple different files from Chrome Desktop & Chrome Mobile. Neither one were blocked, so it's not a new downloader for Chrome browser it seems.

[TorontoBoy]

Here are my AWS IP ranges. As always with AWS there are probably some special IPs that you wish to exclude. I usually need to cut up a range for exclusions such as Pinterest, Facebook, etc.

34.192.0.0 - 34.255.255.255 34.192.0.0/10
52.32.0.0 - 52.63.255.255 52.32.0.0/11
54.184.0.0 - 54.187.255.255 54.184.0.0/14
54.192.0.0 - 54.207.255.255 54.192.0.0/12
54.245.0.0 - 54.245.255.255 54.245.0.0/16

[keyplyr]

@TorontoBoy - More complete AWS range list here. [webmasterworld.com]

Also AWS ranges are somewhere in the Server Farm Thread [webmasterworld.com]

[lucy24]

Only one UA is DownloaderChrome.

Oh, I hadn't noticed that detail. If I cross-check, I consistently get clumps of 3 requests from the same IP, with the UA varying seemingly at random: sometimes all the same, sometimes mixed. Overall, the Firefox/55 UA is a little more frequent. In each case, one of the three requests--the first or the second, never the third--is for the wrong form, www.example.com.

How utterly fascinating.

[TorontoBoy]

If anyone knows or has found out about the origins of DownloaderChrome, please speak up.

[keyplyr]

TorontoBoy that's what the discussion is about. No one is withholding information from you.

My credo has always been that it is up to the botrunner to provide the necessary information for access, otherwise it it blocked. I frankly don't care if a bot "obeys" robots.txt or not. It needs to explicitly convince me that allowing it access to my property benefits me in some way.

[lucy24]

I thought he was being facetious. Will the real DownloaderChrome aka Firefox/55 please stand up.

[TorontoBoy]

SetEnvIf User-Agent DownloaderChrome keep_out

Even banned it can still read my robots, which is open to everyone. It is not accessing anything else. The other UA is too general to pin down.

[keyplyr]

An abuse report has been filed for this UA coming from AWS: [abuseipdb.com...]

[lucy24]

The other UA is too general to pin down.

It may have been one of those short-lived FF releases that was superseded almost at once. I find next to nothing human within this calendar year if I match against the full UA string. Even if I open it back out to any-and-all Firefox/55, I don't see anything since February.

:: detour to check something ::

The robotic UA du jour seems to be Firefox/40.1. One specific full string is nabbed by mod_security; all others go skulking off with a 403. (For deficient headers, not browser version. I'm exceedingly conservative about blocking elderly browsers.)