No referrers from Search Engine

[maurocanzian]

Hi,

in your opinion how many 'no-referrers' come from search engines?

How many are caused by proxy setting or high security level of browser?

[larryn]

100%

Referrers are controlled by the client.

[Dijkgraaf]

Looking at half a months worth of data from my logs, 71% of entries without referers are bots of some type (search engine and hostile bots). This value would probably vary considerably depending on the ratio of human to bot visitors for a particular web site.
I can't quantify whether it is the browser or proxy are hiding the referer for the other entries as there is no obvious indicator except for those that put entries like the XXXX:++++++++++++++++++ in the Referer or http://example.com/ (Unix) in the User Agent.

[edited by: rogerd at 1:18 am (utc) on June 22, 2005]
[edit reason] examplified [/edit]

[larryn]

I mis-understood your question. On second reading, I'm seeing you asking "How many visitors who have no referrer are search engine spiders or other types of robots?".

My experience is that most, but not all visitors without referrers are bots. But that is declining as more and more 'human' visitors are blocking referrer. The best way to determine if something is a 'well behaved' bot is to check for requests for the file robots.txt. The almost as good way is to read the agent/browser string.

As an aside, I read your question originally as "How many visitors who are referred from search engines have no referrer". The reason I stated 100% is because they would still have been referred from search engines.

As reality sets in you will find that you have to look at a series of requests to determine if a specific visitor is a human who typed something in directly, a human who came from a search engine without allowing referrer headers, a well behaved bot who asks for robots.txt, or a rogue bot who just downloads your content.

Good luck!

Larry

[maurocanzian]

My question is:

In your opinion, how many visitors who have no referrer are human visitors coming from search engines?

I can't determine it by log, I'm asking for a general estimate.

Looking at my stats, spiders are about 30-40% of no-referrers. What about the others?

[Receptional]

Maurocanzian,

Maybe you could go and install a javascript based page counter/tracker on your pages for a few days (just find one with a 30 day free trial) and compare the difference in the number of unique visitors the system says.

Javascript tracking is better at identifying humans, because bots tend to ignore the javascript.