Sec-Fetch-* headers

Are there any modern browsers using them?

[blend27]

So we had an outbreak from the glorious B2NETSOLUTIONS range of 104.144.0.0/16, which I missed for some reason.

Using rotating UAs:

Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Firefox/68.0
Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0
iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2)
Links (2.1pre15; FreeBSD 5.3-RELEASE i386; 196x84)
Mozilla/5.0 (compatible; Konqueror/4.5; FreeBSD) KHTML/4.5.4 (like Gecko)
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b4pre) Gecko/20100815 Minefield/4.0b4pre
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.173

And on top of that from 8 rotating IPs from that range.

What stands out is a trio of headers included in each request:

sec-fetch-mode: navigate 
sec-fetch-site: none 
sec-fetch-user: ?1

Are there any modern browsers using them?

[lucy24]

Are there any modern browsers using them?

Yes.

When I saw the subject header I immediately fired up TextWrangler for a search of logged headers (going back only 12 months) and found tens of thousands of Sec-Fetch, most of them without any of the environmental variables that would flag them as unwanted robots. Casual eyeballing even turned up one with a piwik cookie, which is pretty dispositive for identifying a human.

There seem to be four of them, often appearing in a group: -Mode, -Site, -User, -Dest.

That's assuming you mean Sec-Fetch in Title Case. If you're really getting requests with lower-case “sec-fetch” (I didn't find any) those would be robots.

[dstiles]

I also had a spate of B2Net hits on my Windows server, hitting one particular site two days following. The hits were continuous and far in excess of my normal bad traffic. The IPs were already in my traps database but this was ridiculous to I retailiated by sinking about a dozen B2NET IP ranges into the windows ip blocker, a sort of specialised iptables.

[blend27]

I just did an extensive search on headers for sec-fetc-* and it seems to only be present for Chrome Browser. 12 month worth of headers.

One thing in my search instances is that all the occurrences also include Sec-Fetch-Dest header which properly identifies what type of request it is for: image for images(whether the image is called from inline referenced by JS ), document for page...etc

All the requests that came from B2NETSOLUTIONS did not include that particular header, nor they requested images or CSS or executed JS for that matter.

And I am pretty sure that

Links (2.1pre15; FreeBSD 5.3-RELEASE i386; 196x84)

does not support that header :)

[iamlost]

Sec-Fetch-* is currently supported by:
* Android Chrome since version 80
* Android WebView since version 80
* Chrome since version 80
* Edge since version 80
* Opera since version version 67

Sec-Fetch-* is currently NOT supported by: Android Firefox, Android Opera, Firefox, Safari. Nor Internet Explorer :)

Current Working Draft:
Fetch Metadata Request Headers [w3.org] W3C Working Draft, 10 January 2020.

First Working Draft:
Fetch Metadata Request Headers [w3.org] W3C First Public Working Draft, 27 June 2019

Added info:
If you haven’t taken the deep dive into HTTP Headers [developer.mozilla.org] you really should give this a read.

From the above but direct to the subject under discussion: Fetch metadata request headers [developer.mozilla.org]

[blend27]

Thank You IAMLOST!

Hint Hint, Scraper Headers for fake Chrome/84.0.4147.89?

ip: 51.159.37.XX
remote host: 51.159.37.XX
TimeDiff(0)
time: {ts '2021-02-11 09:50:20'}
http_content: 
method: GET
protocol: HTTP/1.1 
Cache-Control: no-cache 
connection: close 
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 
Accept-Language: en-US;q=0.8 
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36 
Accept-Encoding: gzip,deflate 
referer: http://example.com/
pragma: no-cache 
host: example.com
content-length: 0 

[dstiles]

I got a third spate of B2NET hits last night. Later today, all B2NET IPs I can find go into iptables.

[blend27]

--- all B2NET IPs I can find go into iptables.

Good Luck: https://bgp.he.net/AS55286#_prefixes

[edited by: not2easy at 2:19 pm (utc) on Mar 12, 2021]
[edit reason] Fixed # URLs vs. link usability [/edit]

[dstiles]

Thanks! :(

And that's not the only AS they have.

[blend27]

I know @Lucy24 will love this!

I did a test for the past 2 weeks on incoming traffic where the absence of Sec-Fetch-* headers correlate with pretense of Chrome/8[0-9](so far) in UA and ooooooolala!

Tested Reg Chrome 8x+ and when they go to 9x+, Edge(based on Chrome), etc.... nice! And vice versa on older versions of Chrome.

Basically this is in ColdFusion:

<!--- Chrome/8x passes Sec-Fetch-Mode header, all others are fake requests --->
<cfset headers = GetHttpRequestData().headers>
<cfif (find('Chrome/8', headers['user-agent']) or find('Chrome/9', headers['user-agent']))
and
not structKeyExists(headers, 'Sec-Fetch-Mode')>

<cfheader statuscode="403" statustext="Forbidden: Execute access is denied">
<cfabort>
</cfif>

[lucy24]

I know @Lucy24 will love this!

:: jumping up and down with excitement because anything to oblige ::

You’re saying that real, i.e. human, Chrome 80+ sends this set of headers, so if they don’t it’s fake?

:: look at logged headers ::

No, wait, that can’t be right. (And when did Googlebot start calling itself Chrome/8[0-9]? I need to update my At Home with the Robots information, since the answer appears to be “pretty exactly a year ago”. Oops.)

[blend27]

¿si?

From what I see is all validated humanoids at this point that are using Chrome based browser are in fact equipped with Sec-Fetch-*.,

Me Running queries on a year of DataCenter ip requests produce 0.02% with those headers included and none of those are traced to human browser habits.

.Googlebot.tztztz.. are IP Range and RDNS isolated here....

[lucy24]

So header = not human? I’m confused.

When I look at Chrome/8x in logged headers, excluding 66.249, about 1% of them include the Sec-Fetch headers. A good many of that 1%--but by no means all--are clear robots. (I only log headers on page requests, so I can’t generalize about other types of file requests. But I'm going to postulate that a request coming in with a piwik cookie is human.)

Incidentally, it turns out the Chrome/8x Googlebot is a continuation of something they started doing in May 2018, and which I noted at the time. It's a distinct UA, very similar to a human browser, used only for requesting scripts and stylesheets (rarely other non-page content). In January 2020 they changed from unnumbered Safari to Chrome/78, and since then have changed every month or two, apparently matching the current Chrome. So far they’re up to /88.

[dstiles]

This is not always true for other (imitation?) browsers. I have added a test for this to my home-grown log / parser and found this one from at least three Amazon IPs, which is always a bad sign. Note the antique Firefox version.

User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
Sec-Fetch-Site:none
Sec-Fetch-Mode:navigate
Sec-Fetch-User:?1
Sec-Fetch-Dest:document

A Chrome hit from CenturyLink, suspicious in itself as well as being an out-of-date Chrome and Windows XP:

User-Agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36
Sec-Fetch-Site:same-origin
Sec-Fetch-Mode:navigate
Sec-Fetch-User:?1
Sec-Fetch-Dest:document

And from a Clouvider NordVPN:

User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36
Sec-Fetch-Site:none
Sec-Fetch-Mode:cors
Sec-Fetch-Dest:empty

These were all from yesterday's 403 log. On the other hand, several Chrome hits that were banned and showing Sec-Fetch COULD have been genuine but blocked for some other indiscretion.

As a general test I would not be happy with this, but as part of a more expansive test it may help.

[blend27]

-- So header = not human? --

1. The ones that pass Sec-Fetch-* header are supposed to be Chrome/80+

2. If Sec-Fetch-* header is passed and it is not a Chrome based browser it is MOST likely a NOT HUMAN, like in:

iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2)

[lucy24]

OK, gotcha. Whew.

Memo to self: Pore over php docs to figure out how to identify the page the script is actually called from (not necessarily the same as the requested URL, in the case of 403/404/410) as this would tell me which requests were blocked on non-header grounds.

iTunes/4.2 (Macintosh; U; PPC Mac OS X 10.2)

That would be a very, very old Mac if it were human--the Mac equivalent of, say, MSIE 6.

[blend27]

They were using rotating UA route, rotating IPs route and spoofed headers(that were all the same), posting into multiple forms from within the same page, with requests with less than a second. You know, the fun stuff.

Now days with cloud computing, it is possible to initiate these type of requests from multiple IPs. On my end I compare resent set of headers stored in server memory to current request and then if 95+% (usually 99) subsequent requests matching get 'dog-housed' with 403 or dropped in /24 until I get to look at them manually.

Then(just in time) that data is propagated to other servers in the same niche sites. Reason for that id that scrapers and such usually use the same set of sites to....

Kind of like writing Antivirus, but BudLight version of Stoli :)

p.s. Stoli is clear, BudLight is not.

[blend27]

I could confirm that Brave browser also passes sec-fetch headers. Just downloaded one, based on Chrome.

[blend27]

Danger: No Sec-fetch header :(.

Chrome 81 on Android KitCat 4.4.2, this is on a 'slightly' older Samsung Galaxy Tab A. Max version supported is 81.

Cache-Control: max-age=0 
connection: keep-alive 
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 
Accept-Language: en-US,en;q=0.9 
user-agent: Mozilla/5.0 (Linux; Android 4.4.2; GT-N5110) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 
Accept-Encoding: gzip, deflate 
host: example.com 
Upgrade-Insecure-Requests: 1
content-length: 0

[blend27]

RE: Danger: No Sec-fetch header :(.

Actually I made a mistake here I think. It seems that real Chrome and Edge too do not send these headers when the site is accessed via HTTP. They do if connection is encrypted via HTTPS.

Didn't know that until I removed HTTPS redirect in .htaccess and web.config while trying to test something completely unrelated.

[blend27]

I see Firefox 90-Nightly added the these headers too now.

Speaking of which - some MSFT ranges are starting to use latest FF UAs + 1 version.


The usual funny stuff.

[lucy24]

some MSFT ranges are starting to use latest FF UAs + 1 version

How infuriating! How the ### would you block them, short of running an automated process that notifies you the very instant a new version is released so the bad_agent list can be updated.

Incidentally (going back to February) I figured out that I don't need to know what physical page the logheaders function is called from, because I can use Redirect Status instead. (So far I haven't bothered to screen out the 200s, which represent .html or .txt URLs rewritten to php.)

[blend27]

-- How the ### would you block them -

I have one of my virtual machines that I use once a day that when started have 3 browsers open up on start up and updated to the latest version of a browser on a machine, then pointing to specific domain(page) that matches current headers-set to the previous based on documents, .., css, js & image request headers. If something is fishy i get a warning.... call it 'gikieriyo' routine.

Oh, I think I just gave it all away... :)

..unofficial newer versions get home-brewed CAPTCHA challenge.

.. now if I could figure out why am I missing Calcium on 2 of my tomato plants(yellow leaves) - would be totally awesome..

-- So far I haven't bothered to screen out the 200s --

That is where all the juicy stuff is!, all though it does take away from the garden time.

--documents, .., css, js & image request headers --

those file extensions when rewritten via .htaccess or web.config(rules) should pass proper headers as well.

[lucy24]

those file extensions when rewritten via .htaccess or web.config(rules) should pass proper headers as well.

They do. I only log headers on page requests, so the only non-page files whose headers get logged are the blocked ones (logheaders file called by physical 403 page). I do glance at those now and then to make sure legitimate humans aren't getting blocked due to non-page requests not carrying all the same headers.

now if I could figure out why am I missing Calcium on 2 of my tomato plants(yellow leaves)

Just two, while the others appear healthy? That is indeed puzzling--and perhaps more satisfying to solve than any question involving malign robots.

[blend27]

@Lucy

page-request based on IP and server session:

1. page itself
2. files requested from that page:
<< 2.1 XFile.CSS(this one has an image or 2 that are rewritten to specific by extent to be processed logging routine)
<< 2.2 XFile.JS(no pun intended) that also has document.write <img src="boomshakalaka.png"> that is rewritten to a server side template language.

When this is no good:

1. User has cookies turn off and JS turned off(NOscript, UBlock)
2. People that disable CSS and Images <<-- ok, yo site looks like a waffle <-- most do..
3. Speed(no help for Speedy Gonsalez, first 65 seconds explain the theory: [youtube.com...] but then there is a mention of CLOUD at 6:17.... but then one can never go wrong with a commentary at 19:02)


<hr>, :)
My experience might be a bit different; I don't run any external scripts or anything that is pointing to any external sources on a client side of visitors browser.

[lucy24]

2.2 XFile.JS(no pun intended) that also has document.write <img src="boomshakalaka.png"> that is rewritten to a server side template language.

Heh. Years ago, I tracked certain (human) behaviors by sneaking in a couple of image requests. What the visitor saw was a one-pixel transparent gif (i.e. they saw nothing); what I saw was some specific filename indicating some specific behavior.

I actually can't remember what I was tracking, just that this was a quick-and-easy way to do it.