Welcome to WebmasterWorld Guest from 3.80.6.254

Forum Moderators: Ocean10000

Message Too Old, No Replies

DownloaderChrome

     
7:19 pm on Apr 26, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15928
votes: 884


Does anyone know what the heck this is?

Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
It doesn’t appear to be a human addon. They’ve been coming around since the beginning of the month, always with that exact UA. I haven't noted the IPs more exactly, but they're all in 34, 52, and 54 so presumably AWS.

Curiouser still: Their only request, ever, is for robots.txt, always with a referer in one of these four forms (the site is http://example.com):
http:// example.com
http:// www.example.com
http://example.com/robots.txt
http://www.example.com/robots.txt
Yes, that's an actual space in the first two, not an artifact of my log-wrangling. I double-checked.

As you can imagine, googling this exact name is not a fruitful endeavor.
7:20 pm on Apr 26, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


Looks like a downloader for Chrome :)

Actually, it's probably just a bot scraping data. A web search returns server logs with common mentions of this UA.

Please post the IP range when you report UA. That is relevant information to determine who/what this really is.

[fix typo]

[edited by: keyplyr at 7:26 pm (utc) on Apr 26, 2018]

7:25 pm on Apr 26, 2018 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 30, 2002
posts: 2661
votes: 103


Could be a bot but there's a report of Adware of that name in Bleepingcomputer.

Regards...jmcc
8:48 pm on Apr 26, 2018 (gmt 0)

Preferred Member from CA 

Top Contributors Of The Month

joined:Feb 7, 2017
posts: 579
votes: 60


52.41.56.255[02/Apr/2018:12:25:15 GET /robots.txt HTTP/1.12001285 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.201.164.157 [05/Apr/2018:04:48:39 GET /robots.txt HTTP/1.12001285 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.217.117.174 [05/Apr/2018:19:50:02 GET /robots.txt HTTP/1.12001285 http://www.example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.211.123.247 [06/Apr/2018:01:05:15 GET /robots.txt HTTP/1.12001285 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.245.177.115 [11/Apr/2018:07:13:11 GET /robots.txt HTTP/1.12001414 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.187.172.150 [11/Apr/2018:22:32:19 GET /robots.txt HTTP/1.12001414 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.212.173.132 [12/Apr/2018:05:18:18 GET /robots.txt HTTP/1.12001414 http://example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
34.212.173.132 [12/Apr/2018:05:18:18 GET /robots.txt HTTP/1.12001414 http://www.example.com/robots.txtMozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36
54.202.187.111 [24/Apr/2018:05:24:28 GET /robots.txt HTTP/1.12001478 http:// www.example.com Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) DownloaderChrome/62.0.3202.75 Safari/537.36

All are Amazon AWS IP, which I ban, with the exception of robots and my 403. I could not find the origin in a Google or Baidu search. There is nothing on Git. Many other sites have also seen this UA and have reported so. All my hits have been with different IPs from AWS, and with one exception, never repeated. They don't look at any other resource other than robots.txt.

Curious. This is a home grown bot of unknown origin and not a copycat.
10:13 pm on Apr 26, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15928
votes: 884


never repeated
Do you mean from one day to another, or within the same day? Mine tend to come in clumps of 2 or 3 from the same IP in rapid succession, and when this is the case they toggle the pseudo-refererer, adding or removing the www. Technically they are probably changing the host itself, and setting the referer to match, but I don't log headers on 301 responses--mainly because I don't know how to--so I can't be sure.

But yes, now that you mention it, it's a new IP every time, or every clump.

If they were requesting anything other than robots.txt they would be blocked due to one absent header.
10:32 pm on Apr 26, 2018 (gmt 0)

Preferred Member from CA 

Top Contributors Of The Month

joined:Feb 7, 2017
posts: 579
votes: 60


I posted all my sightings from my 2018 April log. If you wish I could post my March sightings.

They clump only in 2s, the same IP per day, a cycle (clump!) every 3-4 days, but 2 different UAs. Only one UA is DownloaderChrome. The other UA is: "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0". Either could be first, but usually it is DownloaderChrome. Only once was DownloaderChrome used 2 times and the other not used at all. IPs are rarely repeated between cycles for the month of April.

The first referrer is always http://example.com/robots.txt. The second referrer is always http://www.example.com/robots.txt. They seek nothing else, and if they did I serve them 403s.
11:13 pm on Apr 26, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


All are Amazon AWS IP
This is why we need the IP rangs(s) in the 1st post. That pretty much confirms it's not human.

Just to make sure, I blocked the UA, then downloaded a couple different files from Chrome Desktop & Chrome Mobile. Neither one were blocked, so it's not a new downloader for Chrome browser it seems.
11:26 pm on Apr 26, 2018 (gmt 0)

Preferred Member from CA 

Top Contributors Of The Month

joined:Feb 7, 2017
posts: 579
votes: 60


Here are my AWS IP ranges. As always with AWS there are probably some special IPs that you wish to exclude. I usually need to cut up a range for exclusions such as Pinterest, Facebook, etc.

34.192.0.0 - 34.255.255.255 34.192.0.0/10
52.32.0.0 - 52.63.255.255 52.32.0.0/11
54.184.0.0 - 54.187.255.255 54.184.0.0/14
54.192.0.0 - 54.207.255.255 54.192.0.0/12
54.245.0.0 - 54.245.255.255 54.245.0.0/16
11:34 pm on Apr 26, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


@TorontoBoy - More complete AWS range list here. [webmasterworld.com]

Also AWS ranges are somewhere in the Server Farm Thread [webmasterworld.com]
4:11 am on Apr 27, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15928
votes: 884


Only one UA is DownloaderChrome.
Oh, I hadn't noticed that detail. If I cross-check, I consistently get clumps of 3 requests from the same IP, with the UA varying seemingly at random: sometimes all the same, sometimes mixed. Overall, the Firefox/55 UA is a little more frequent. In each case, one of the three requests--the first or the second, never the third--is for the wrong form, www.example.com.

How utterly fascinating.
10:57 pm on Apr 27, 2018 (gmt 0)

Preferred Member from CA 

Top Contributors Of The Month

joined:Feb 7, 2017
posts: 579
votes: 60


If anyone knows or has found out about the origins of DownloaderChrome, please speak up.
11:45 pm on Apr 27, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


TorontoBoy that's what the discussion is about. No one is withholding information from you.

My credo has always been that it is up to the botrunner to provide the necessary information for access, otherwise it it blocked. I frankly don't care if a bot "obeys" robots.txt or not. It needs to explicitly convince me that allowing it access to my property benefits me in some way.
1:04 am on Apr 28, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15928
votes: 884


I thought he was being facetious. Will the real DownloaderChrome aka Firefox/55 please stand up.
2:21 am on Apr 28, 2018 (gmt 0)

Preferred Member from CA 

Top Contributors Of The Month

joined:Feb 7, 2017
posts: 579
votes: 60


SetEnvIf User-Agent DownloaderChrome keep_out

Even banned it can still read my robots, which is open to everyone. It is not accessing anything else. The other UA is too general to pin down.
3:30 am on Apr 28, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


An abuse report has been filed for this UA coming from AWS: [abuseipdb.com...]
7:01 am on Apr 28, 2018 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15928
votes: 884


The other UA is too general to pin down.
It may have been one of those short-lived FF releases that was superseded almost at once. I find next to nothing human within this calendar year if I match against the full UA string. Even if I open it back out to any-and-all Firefox/55, I don't see anything since February.

:: detour to check something ::

The robotic UA du jour seems to be Firefox/40.1. One specific full string is nabbed by mod_security; all others go skulking off with a 403. (For deficient headers, not browser version. I'm exceedingly conservative about blocking elderly browsers.)