Mandatory Web Browser Headers?

[JorgeV]

Hello-

I am wondering to know if there is a list of "mandatory' headers sent by browsers?

For example, I see a significant number of requests without "accept" , "accept-encoding" or "accept-language" . I am trying to figure out if these requests are legitimate, or from bots.

Thank you,

[lucy24]

Depends what you mean by “mandatory”. Yes, there are absolutely some headers that all human browsers send, and sometimes even header values that are different for humans than for robots. And, conversely, there are some headers that only robots send. I moved over to header-based access controls several years ago, making life far far easier.

But this is an area where it's best to do your own individual research. See what your humans always send; see what the robots--including authorized ones, like search-engine crawlers--often omit. Start by looking at unambiguous robots--the ones that request only a page, with no supporting files.

And, just like user-agents (remember when anything starting in “Mozilla” could be safely flagged as human?), you can't just check it once and for all and never think about it again.

[not2easy]

There have been a few discussions on a related topic that can shed light on your question. Some webmasters keep a log of all headers, specifically to know whether they are from bots or humans.

How it works: [webmasterworld.com...]

The whys and whats are here: [webmasterworld.com...]

There are other threads specifically discussing bot headers, the site search in the upper right can help with that.

Huh. Crossposting. Edited to add- lucy24 is right, the logging is to help you in your research. It isn't a copy/paste fix that one can set and forget.

[JorgeV]

Thank you!