HTTP header fields

ACCEPT and ACCEPT-CHARSET

[dstiles]

Prompted to look into these again following some odd traps made on the ACCEPT header.

Chrome/31 seems, at least in some circumstances, to be either not sending this header or (less likely) sending it as blank. Until now this header has been a reasonable "human" indicator, apart from a few bad proxy setups. This is the only browser I've seen so far and most (but not all) of the traps are in the Apple IP range 17.142.0.0/16. I wonder if it's some mobile gizmo.

Are there any thoughts about this?

Looking up comments for this header (none found so far) I came across notes about the CHARSET header, which now appears to be defunct. Although maybe not on google chrome...

[hsivonen.fi...]

The reasoning behind the change seems valid.

[trintragula]

I appear to have trapped a handful of python-requests/1.2.3 in 17.142 in November. They're the only things I've stopped from the whole Apple range. The last bot pretending to be Chrome/31 that I've stopped was ten days earlier, but wasn't the Mac version, and not from the Apple range. Chrome/31 is old now, and I'm not seeing much traffic from it.
My default assumption would be that if the headers are wrong the UA is faked...

I'd been assuming that the specifics of header scrutiny are not to be discussed openly, though I can see arguments for doing so and for not.

[lucy24]

I'd been assuming that the specifics of header scrutiny are not to be discussed openly

It kinda depends on how many botrunners read this forum and take action accordingly, doesn't it? Heaven knows we still get plenty of requests like (grabbing the first specimen I find)

2014-12-17:16:04:58
IP: 12.34.45.78
Host: example.com
Accept: */*
Expect: 100-continue
Connection: close
----

2014-12-17:16:04:59
IP: 12.34.45.78
Host: example.com
Connection: close

(and that was all she wrote) ... which you wouldn't think could get through anywhere. Even the faviconbot now sends a UA header, albeit a ridiculous one (FF6, last I checked).

Incidentally, what on earth is an Expect: header? I was just looking for something without the UA header and this came bobbing to the surface* at the same time. Poring over the horse's mouth [w3.org] leaves me no wiser. Close study of access logs tells me that this particular header came with a POST request-- which is itself automatically blocked unless they're getting the Contact page-- but it isn't an inherent feature of POST.

Don't know about most people, but I include the log-headers footer even in my 403 page. This gives a little further insight into visitors who would normally be out of sight, out of mind. There's even the occasional image request, so I get an idea what those headers look like.


* "Cream rises. So does scum."