Headers Trivia & Fun

Who could tell me the problems with these headers sent to my site?

[blend27]

For those who wake up early, so to speak...

So here is one of those Super Smart Scraping scripts that I mentioned in [webmasterworld.com...]

Who could spot the irregularity in these request headers.

Scraper(178.157.64./24) -- got to the first page(was tripped into a soft 403 - page served had about 60 random links on it) and visited every single link with in 31 seconds, with similar headers! It is not the speed though that we are after.

The Monsignor ate all cookies it was fed and returned a valid one at each request. So, it is not that either....

JSON:

"headers": {
"sec-ch-ua-mobile": "?0",
"user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.5",
"sec-ch-ua": "\".Not/A)Brand\";v=\"99\", \"Google Chrome\";v=\"103\", \"Chromium\";v=\"103\"",
"host": "www.example.com",
"X-REWRITE-URL": "/",
"DNT": "1",
"connection": "keep-alive",
"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Origin": "https://www.example.com",
"Accept-Encoding": "gzip, deflate;q=1.0, *;q=0.5",
"referer": "https://www.example.com",
"cookie": "valid_cookie_value",
"sec-gpc": "1",
"sec-ch-ua-platform": "\"Linux\"",
"content-length": "0"
}

Hint: It is missing a bunch and one is set to a value that should not be allowed without the one that is missing.

Have at it!
:)

[blend27]

OK, I will do an easy one:

1. Sec-Fetch-Mode(Should be set to "navigate") Header is missing along with a missing Header Sec-Fetch-Dest(because it is request to a document/site-root should be set to "navigate").

https://chromestatus.com/feature/5155867204780032

https://web.dev/fetch-metadata/
Browser compatibility #
Fetch Metadata request headers are supported as of Firefox 90 and as of Chrome 76 in all Chromium-based browsers.

https://wicg.github.io/ua-client-hints/#spam-filtering-bots-use-case

https://www.w3.org/TR/fetch-metadata/

2. "Origin": "https://www.example.com" - without Sec-Fetch-Mode(which should be "cors" in this case)

"Origin" header is included is when request is made via FETCH/XHR/AJAX type requests.

Luckily for me all requests that originate from within this website have an extra header inserted via AJAX Setup

$.ajaxSetup({
 headers: { 'ajax-x-from': 'UUID-Assigned-To-Session' }
});
$.post( endPointURI,..................

So the bugger did not make it, well only to to toInt(404-1) :)......

There is a lot of info in those link I added.
--------------------------------------------------------------------------------
P.S. I was previously mistaken that the Chrome version that started to include SEC-Fetch-* headers was 79, it is actually 76. And now i will have to email all those users that got presented with the Captcha an apology. I could get those from IP Address right?

[lucy24]

I have never gone to the trouble of learning anything whatsoever about that annoying cluster of Sec-blahblah headers that have become so prevalent in recent years. Perhaps I should?

[blend27]

-- Perhaps I should? --

Perhaps, it is a lot of fun when you can.

1. Sec-Fetch-Mode(Should be set to "navigate") Header is missing along with a missing Header Sec-Fetch-Dest(because it is request to a document/site-root should be set to "document").

^^ added correction.

[blend27]

u=1600.20000.51142847.1 443 - 66.249.80.166 Mozilla/5.0+(en-us)+AppleWebKit/537.36(KHTML,+like+Gecko;+Google-Adwords-DisplayAds-WebRender;)+Chrome/109.0.5414.101Safari/537.36

Google-Adwords-DisplayAds-WebRender = Resolution is at 1600 x 20000 pixels.

None of other bots request tracking image.

[blend27]

@Lucy,

Matomo(was it {PIWIK} before?) has a pretty cool lib called ClientHints [github.com...] .

There are all kind of hints that could be used on Headers inspection routine....

[lucy24]

was it {PIWIK} before?

Yes, in fact all of the files and directories on my site are still called piwik. Matomo doesn't seem to mind.