Phasing out User Agents

What will they look like and how will they affect us

[dstiles]

Google has announced plans today to phase out the usage of user-agent strings in its web browser Chrome. ... UA strings have been used by online advertisers as a way to track and fingerprint website visitors. ... User-Agent sniffing is an abundant source of compatibility issues, in particular for minority browsers ... Google said it plans to phase out the importance of UA strings in Chrome by freezing the standard as a whole.

Ref: [zdnet.com...]

The other main browsers are to follow google in this. I have yet to discover to what degree the UAs will be deprecated - I assume there will be a degree of UA remaining but what will it tell us? The report suggests there will be a generic UA denoting, at most, Desktop or Mobile. If it omits the version number how can we ban the older browsers, for example. Presumably bots will still have an identifier?

The fact that it will identify itself as Chrome (eg) does not help in one of its aims, that of minority browser compatibilty issues. Vivaldi has already decided to re-identify itself as Chrome because web sites are rejecting the unfamiliar name Vivaldi.

The deprecation of the UA string mechanism is part of a push at Google to improve privacy on the web, but without killing online advertising, the lifeblood of most free websites today.

So advertisers are ok, then. Although G is going to block third party cookies - which others do by default.

UA strings in Chrome will be replaced with a new mechanism called Client Hints ... through which websites can request information about a user, but without "the historical baggage and passive fingerprinting surface exposed by the venerable `User-Agent` header,"

There is information on the Hints mechanism at [wicg.github.io...]

How easy would it be for bad bots to simulate this? In fact, the specification seems to supply all the elements of a UA but using separate variables, which to my mind makes it more difficult to track.

I have already noticed Sec- fields amongst some MS browser headers (eg Sec-Fetch-Site). This does not gel with the G proposal names, though. Are they going to be differrent per browser? Presumably information about these Hints will soon become more readily available complete with lists and layman-type explanations.

Does anyone already have a nascent mechanism in place to detect and use these Hints?

[not2easy]

There was an announcement here: Google Wants to Freeze and Unify Browser User Agent String that talks more about unifying than removing the UA: [webmasterworld.com...]

[dstiles]

Sorry, I don't monitor analytics. I think this has a broader effect than analytics, anyway.

[Kendo]

phase out the usage of user-agent strings in its web browser

This would be disastrous. Without UA web pages cannot cater for the different browser quirks and mobile phone compatibility goes out the window. Chrome is not a good example and never has been. Compliance with WC3 standards might help if they ever bothered.

Then you have web browsers that have the capacity of changing UA for compatibility on some sites, and then you have a plethora of browser extensions that can falsify UA completely.

[lucy24]

What, exactly, do they mean by “phase out UA string”? If they mean “stop sending the User-Agent header” that would be a nifty way to kill their browser* when users discover that their favorite sites simply won’t let them in.

Without UA web pages cannot cater for the different browser quirks and mobile phone compatibility goes out the window.

The UA string has nothing to do with responsiveness, and “browser quirks” are their problem, not ours. If lack of detailed UA information forces websites to become less fancy, that can only be a good thing. (My elderly iPad crashes on at least half of the sites it tries to visit. It does not crash on mine.)


* “Chrome, meet Netscape. Netscape, meet Chrome. I’m sure you guys will have a lot to talk about.”

[Kendo]

The UA string has nothing to do with responsiveness

It is the UA that we detect so that we can change code to suit the browser. Not all browsers support all JavaScript calls and it wasn't long ago that I had to make adjustments for Chrome browser because it was rendering stuff with a 1 pixel difference to what it looked like in other browsers.

I already see too many web sites that only function correctly in Chrome browser.

[tangor]

Or ... this could be just one more step to "one size fits all" g style.

After all, if you can't tell who is sending stuff to you...

YMMV

Information is how the world goes round. Remove some of that info and your world revolves a lot slower. And if only ONE source of that information is out there you are tethered to that source.

Or it could mean g was intent on making all THEIR UAs look the same. :)

I'll bring the popcorn as it appears the main feature is about to start.

[notriddle]

This news article is a waste of space. Just read the actual announcement and the spec. It's not that long. [groups.google.com...]

They've answered most of your questions:

This would be disastrous. Without UA web pages cannot cater for the different browser quirks and mobile phone compatibility goes out the window. Chrome is not a good example and never has been. Compliance with WC3 standards might help if they ever bothered.

The User-Agent string is being replaced with Client Hints. The difference is that you have to send a header requesting a hint, and then the browser sends the hint's value in the following requests. This way, instead of just passively providing the information in every request, you can tell whether a site uses certain information or not. This also means that only the information that the site asks for actually gets sent, so the request doesn't get bloated with a bunch of information that they don't actually use.

How easy would it be for bad bots to simulate this? In fact, the specification seems to supply all the elements of a UA but using separate variables, which to my mind makes it more difficult to track.

There's a stateful aspect to this, so it'll probably be wrongly-implemented by at least some bad bots.

Then you have web browsers that have the capacity of changing UA for compatibility on some sites, and then you have a plethora of browser extensions that can falsify UA completely.

The only difference between the old User-Agent header and the new Hints system is that the website has to ask for hints, instead of just getting the information by default. A browser extension that can rewrite outgoing requests should be able to falsify Client Hints if that's what you want.

Or it could mean g was intent on making all THEIR UAs look the same. :)

According to [groups.google.com...] it seems like their plan is as follows:

* First, they'll write a warning to the console if you access the navigator.userAgent from JavaScript.

* Next, "freeze" the Chrome and Blink version in the UA string. This means that the version number will become a lie. Nothing will change; what will happen is that the UA header stops changing. It'll still allow you to distinguish Chrome from Firefox.

* Finally, they'll remove the operating system and device model from the UA string. Again, you'll still be able to distinguish Chrome from Firefox, and both will still be different from IE11. I think Edge will probably just start sending the exact same UA string as Chrome, though.

They haven't vocalized any plans to actually remove the UA string. Just "deprecate" and freeze it. AFAICT, these plans have no intent to make mobile and desktop UAs the same, so anyone who switches between mobile and desktop mode based on the UA will still be fine. The main change is removing the operating system from the UA.

[lucy24]

They've answered most of your questions:

Nope, they’ve answered none of my questions, since all I get (from both links) is a Google Groups login page.

the website has to ask for hints, instead of just getting the information by default

“Give me a hint: Are you human?”

I’m sorry, but this sounds entirely senseless.

Current system:
Visitor: “Can I have this page? My name is ABC and I live at XYZ.”
Website (after checking that ABC and XYZ are authorized): “Sure, here you go.”

Proposed system as I understand it:
Visitor: “Can I have this page? I live at XYZ.”
Website (after checking that XYZ is authorized): “Uhm... Who are you?”
Visitor: “My name is ABC.”
Website (after checking that ABC is authorized): “OK, here you go.”

How does this not result in a doubling of response time, to say nothing of increased bandwidth from that extra request-and-response pair?

[topr8]

How does this not result in a doubling of response time, to say nothing of increased bandwidth from that extra request-and-response pair?

that did occur to me too ... it seemed to me that an extra request & response has to make the whole process slower.

it also seems to me that if the server can request various other headers (eg hints) then why not just send them all in the first place? what would the reason be for not doing so? what's to stop all servers just requesting all the 'hints' - and i imagine a lot will.

side note ... personally for my own sites (i don't do any nefarious tracking through UA or anything else) ....
what i'd really like to know when the user first sends a request, is ...

1. connection speed (or bandwidth limitations) eg. i'm on a slow mobile connection or i'm on a fast connection however i have a low data allowance! .... this way i can serve images appropriately.

2. mobile or desktop - i do serve responsive pages, but i could trim a bit of content if i knew for sure i was serving to mobile.

3. actual screen size (not resolution - some phones have bigger resolutions than desktops) ... i don't really care what anyone says - there seems no point to me, serving a very high res image to a phone, there is no way you can appreciate it in the same way as a user with a 30 inch widescreen desktop monitor can (or indeed if you are viewing on a 50 inch tv)!

[dstiles]

I would want ALL the possible hints from a browser (or bot) in order to make access decisions. I assume most active webmasters would, too. The question is: how does one do this?

And someone needs to specify absolutely what Hints can be discounted as bad bots. Which I don't think anyone can - bots will continue to falsify as before - so we're back to the old method of trial and error.

And how does this compare to the Sec-Fetch-Site and similar headers already being issued - and how are those being delivered - on request (in which case my (apache) server is already asking for them) or on initial contact, along with (eg) UA.

And why has G not yet applied Hints to their bots? Or will it ever? Or are we just missing them because we do not know how to get them?

[lucy24]

And why has G not yet applied Hints to their bots? Or will it ever? Or are we just missing them because we do not know how to get them?

They definitely haven't started sending any new request headers, unless they did so after midnight (PST) today. fwiw, their last header change was in early March 2019, when they added Amp-Cache-Transform to those requests that come with If-Modified-Since.

[blend27]

Should not be difficult to figure that out when it is implemented...

But!

I am not sharing my code with anybody :)

hint-hint.. :)

[notriddle]

“Give me a hint: Are you human?”

*bleep* *bloop* sure, I'm totally not a robot

How does this not result in a doubling of response time, to say nothing of increased bandwidth from that extra request-and-response pair?

I, personally, don't plan to use C-H at all in the near future for exactly this reason, and it's plausible that I never will. The bots that I want to block are the same bots that would just lie anyhow. I pretty much only block by IP address.

Nope, they’ve answered none of my questions, since all I get (from both links) is a Google Groups login page.

Blame this forum software's redirect system for stripping off the # anchor from the URL I pasted.

Anyway, here's the actual spec. No anchors, so the link should actually work this time. [wicg.github.io...]

[ClosedForLunch]

One issue is that the server has to ask for the hints, they are not provided by default.

The server can only ask for the hints when it serves the first page requested by a user.

If a user requests a second page, only then are the hints sent to the server.

So, a server can only act on hints from the second page request onwards, not the first.