unknown

[keyplyr]

There used to be several publicly available GET scripts where the user would fill-in the text fields regarding the UA they wished to impersonate. If left blank, at least one of these fields would display as "unknown" in the server logs. I started blocking "unknown" and until recently, never looked back.

Now I'm starting to see "unknown" in mobile UAs:

Mozilla/5.0 (Linux; U; Android 4.2.2; en-us; unknown Build/JDQ39) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30

Anyone know if this is valid?

[tangor]

Valid or not I can't say. But it is ambiguous enough that I block it anyway. YMMV.

[keyplyr]

The concern comes from:
• The IP was a known mobile ISP
• The referrer was a SE with a long tail search query
• After getting the initial 403, the next request was for favicon

This would be uncommon for most bots, so sitting on the fence until I get more info.

[lucy24]

After getting the initial 403, the next request was for favicon

Is that typical for otherwise-similar Androids? (Got a vague impression that yes, they do ask for the favicon, while iThingies don't.)

You've probably already noticed that androids (real ones) can be severely wonky w/r/t referers of non-page files, which doesn't help when you're trying to figure out if it is real.

:: detour to logs ::

Huh, never noticed that. Where I see a lot of "unknown" is in search-engine referers-- the long complicated ones-- that say things like
client=ms-android-samsung&source=android-unknown
client=ms-android-sprint-us&hl=en&gl=us&source=android-unknown
etcetera. So yup, there's definitely an element of "it looks like an android, but more than that I cannot say".

I'd try an extra step, like blocking "unknown" only if the UA string doesn't contain "Android". (Environmental variables are great for this, because you can do a long complicated test once, and then have the results ready and waiting for any mod that needs it, in any combination you see fit.)

[keyplyr]

Android behavior may seem "wonky" due to the fact this attribute is present in browsers of hundreds of different model phone & tablets made by as many manufacturers... while the iphone attribute occurs only in iphones & ipads AFAIK.

But are you sure iphone doesn't ask for favicon after getting a 403? Is it the response code itself or the type of document served? I reference the favicon from the head of all pages, even my custom 403. In addition to the root directory, I also keep a favicon .ico,.png &.gif in most every other directory.

Gonna just allow "unknown" for a while and keep an attentive watch.

[lucy24]

are you sure iphone doesn't ask for favicon after getting a 403?

I was all set to say that iDoodads don't ask for the favicon at all, they ask for the apple-touch-icon instead. But this turns out not to be the case; sometimes it's one, sometimes the other. In fact I can't figure out how the system decides which to ask for. I couldn't find anyone asking for both.

Finding blocked human iOS users is tricky; the vast majority are botnets, meaning they only request pages. Even those infected Russians aren't probative, because the robot typically only asks for supporting files that are explicitly named on the page, which for me excludes any and all icons.

Conclusion after poring over logs: most blocked humans using iOS don't ask for any icon at all, whether favicon or apple-touch-icon.

[keyplyr]

Conclusion after poring over logs: most blocked humans using iOS don't ask for any icon at all

Good... now I just need to know whether Android does it :)

It would actually be nice to be able to distinguish a real Android visit from a poser using something as simple as this.

[lucy24]

It would actually be nice to be able to distinguish a real Android visit from a poser using something as simple as this.

If only we could distinguish them ahead of time instead of figuring it out after the fact by poring over logs :(

:: detour to raw logs again ::

I think 403'd Androids do ask for the favicon. That's based on picking out some of the few that are neither part of a botnet (is the Android inordinately infection-prone, or is "Android" simply the User-Agent du jour among bot-runners?) nor googleweblight (which never gets the favicon), just ordinary-looking humans. The ones that also request stylesheets and piwik and so on.

[keyplyr]

Actually I have not met any Android botnets (yet.) Haven't had any in a few months, but most have been Linux, MSIE, Mac Safari or assorted in the past

[aristotle]

Some android browsers don't give your server information about the (real human) visitor, such as the referer.

For example, if a human comes from Google's search results by clicking one of the results, the android browser will leave the referer blank. It's traffic from Google, but you wouldn't know this, because your logs don't show where it came from.

[blend27]

Don't forget:

Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.6.1 Safari/534.34
Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) PhantomJS/1.9.1 Safari/534.34

and such. They do get all supported files when they Run.