Welcome to WebmasterWorld Guest from 3.92.92.168

Forum Moderators: phranque

Mandatory Web Browser Headers?

     
5:26 pm on Oct 17, 2019 (gmt 0)

Junior Member

Top Contributors Of The Month

joined:Aug 30, 2019
posts:145
votes: 28


Hello-

I am wondering to know if there is a list of "mandatory' headers sent by browsers?

For example, I see a significant number of requests without "accept" , "accept-encoding" or "accept-language" . I am trying to figure out if these requests are legitimate, or from bots.

Thank you,
5:44 pm on Oct 17, 2019 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15934
votes: 889


Depends what you mean by “mandatory”. Yes, there are absolutely some headers that all human browsers send, and sometimes even header values that are different for humans than for robots. And, conversely, there are some headers that only robots send. I moved over to header-based access controls several years ago, making life far far easier.

But this is an area where it's best to do your own individual research. See what your humans always send; see what the robots--including authorized ones, like search-engine crawlers--often omit. Start by looking at unambiguous robots--the ones that request only a page, with no supporting files.

And, just like user-agents (remember when anything starting in “Mozilla” could be safely flagged as human?), you can't just check it once and for all and never think about it again.
5:50 pm on Oct 17, 2019 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4558
votes: 363


There have been a few discussions on a related topic that can shed light on your question. Some webmasters keep a log of all headers, specifically to know whether they are from bots or humans.

How it works: [webmasterworld.com...]

The whys and whats are here: [webmasterworld.com...]

There are other threads specifically discussing bot headers, the site search in the upper right can help with that.

Huh. Crossposting. Edited to add- lucy24 is right, the logging is to help you in your research. It isn't a copy/paste fix that one can set and forget.

5:54 pm on Oct 17, 2019 (gmt 0)

Junior Member

Top Contributors Of The Month

joined:Aug 30, 2019
posts:145
votes: 28


Thank you!