Trying to identify some UA

[QuaterPan]

I'd like to know if these UA are legitimate :

- com.google.GoogleMobile/51.0.0 iPhone/10.2.1hw/iPhone9_3

- Android App v6.87.1; Model: SM-G610M; Android SDK: 24;Connection: WiFi; Locale: es_US;

- Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0;+http://duckduckgo.com) (but from an amazonaws IP range)


Thank you.

[lucy24]

Look legit to me--at least the first and third. I’ve never seen the string “Android App” or “Android SDK”.

I too have been seeing the com.google.GoogleMobile for a while in isolated image requests (that is, not associated with pages). Possibly image search using an app? Closer look says they appeared pretty suddenly in mid-February 2017.

I see the DDG faviconbot from two different IPS, down to the last /32 (commercial ranges, so no need to obfuscate):
107.21.1.*
54.208.102.**
Both AWS; the first is more common. Behavior is identical between the two, so I've no reason to think the second one is bogus.

- - -

[edited by: keyplyr at 6:14 pm (utc) on Jun 15, 2018]
[edit reason] Obscured private IP address [/edit]

[TorontoBoy]

Here are some from my log:

com.google.GoogleMobile/41.0.0 iPhone/10.2.1 hw/iPhone5_1

It seems legit.

Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)

Yes, I had to make a hole in my AWS range ban for this at 107.21.1.0/24, but it seems legit.

I don't get the second one so cannot comment. What does it do?

[QuaterPan]

Thank you lucy24 & TorontoBoy.

For the first, I am surprised that the borwser is not identifying itself , and to see the words GoogleMobile and iPhone associated.

The second, I see it often (several times per hours), from AWS range (it looks like you block AWS so it might be why you are not seeing it). It tries to fetch image (try only , because it's automatically blocked by my rules). I guess it's not legitimate so.

For the third, I was surprised to see it coming from an AWS range, and not DDC 's own IP.

Thank you for your answers.

[TorontoBoy]

I have never seen DDG with their own range. They use AWS, as does Pinterest. This is their decision. I block all of AWS and then poke holes into it for the likes of these two.

[keyplyr]

I'd like to know if these UA are legitimate :

Define "legitimate."

Those UAs would be common, but bots often fake UAs to access your files undetected. You'll need to do more research to determine whether these are requests you wish to allow or not.

This UA:

com.google.GoogleMobile/51.0.0 iPhone/10.2.1hw/iPhone9_3

is an image scraper. Most everyone I know blocks it. It is *not* Google. It is an app that can be used for Android or iPhone (iPhone in this UA) that scrapes images from websites and puts them in an image gallery to be stored or shared.

Don't forget, developers & bot runners can name their agent anything they want, and often use benign or misleading names to gain access to your files.

[Leosghost]

^^^^^ This ( final paragraph , two lines ) should be pinned to the top of every thread in "spiders"..

[lucy24]

is an image scraper.

Ah, thanks, I had no idea what they're for. I think I thought they were one of those bogus pre-fetchers, where the image the search engine preloads isn't the image the user sees, so why bother to serve them a 50k file that most of them will never see.

I started investigating this a little more closely, and was all set to construct a complicated RegEx when I noticed that all blahblahGoogleMobileblahblah requests come away with a filesize of 376, meaning that they are running into a RewriteRule I'd forgotten all about. That's the logged size of my administrative gif, and hence less work for the server than any type of 403, with the same practical result.

Some poring over htaccess reveals

RewriteCond %{HTTP_USER_AGENT} ^(rarely\ used|com\.google\.GoogleMobile)

Huh. I thought I was kidding, but there really was a RewriteRule I’d forgotten all about.

No sign of the UA before February 2017, and the rewrite seems to have gone into effect within weeks. I guess my memory is growing shorter :(

:: wandering off to deal with newly discovered quirks and anomalies in personal site's htaccess, which I haven't been looking at as carefully as I ought ::

[TorontoBoy]

SetEnvIf User-Agent "com\.google\.GoogleMobile" keep_out

I really thought this was from Android, no?

[keyplyr]

From last year: [webmasterworld.com...]


- - -

I’ve never seen the string “Android App” or “Android SDK.

An SDK is a Software Development Kit. Developers for Android, or iPhone, etc use these sets of files to develop their App.

[QuaterPan]

Define "legitimate."

Yes, sorry. I meant do these UA exist in the Nature. Regardless of if they are faked, or on the wrong IP range, I wanted to know if these UA were officially attached to something particular. Like for example your answer about "com.google.GoogleMobile" is the kind I was looking for.

Thank you all.

[QuaterPan]

Yes, I had to make a hole in my AWS range ban for this at 107.21.1.0/24, but it seems legit.

In my case, hits from DuckDuckGo-Favicons-Bot, also come from 54.208.102.**


- - -

[edited by: keyplyr at 6:15 pm (utc) on Jun 15, 2018]
[edit reason] Obscured private IP address [/edit]

[keyplyr]

Please do not post specific IP addresses even if they may be commercial. Only post ranges to avoid compromising the privacy of the owner.