Human visits and robots visits

How to differentiate them?

[Eugenios]

Perhaps it is a very basic question, but how can I differentiate them without using tracking gifs with javascript or known robots lists?

I don't know if there is any parameter that can be considered related to a human visit other than the user agent. It is impossible to keep an up to date listing of browsers ua or robots ua (search engine bots, site downloaders, email harvesters, etc.).

Is here any trick to recognize the human visits? Perhaps calculating the time spent on a page?

Any input will help

Thanks,

Enrique

[larryn]

Enrique,

What I do is three-fold: first I check for visitors that request 'robots.txt'; next I check the user agent (browser) for telltales such as 'bot' or 'spider' or 'crawler', etc; finally, I look for a very fast series of requests for pages.

Hope that helps,

Larry

[Eugenios]

Thanks Larry,

Looking for a fast series of pages could be a good one.

Perhaps checking if an IP requested more than 30 or 40 pages in one session could be another way?

So far the way o recognizing non human visits is:

1) If the ip checks for robots.txt
2) If the ua has certain keywords
3) The speed of requests
4) The quantity of requests in a single session

Any other?

Enrique

[Sanenet]

The regularity of requests. If it's the same length of time between requests, its a bot, even if it didn't grab many pages.

Also, if it's grabbing images, includes, etc.

[pmkpmk]

I recently stumbled across a nice little program by the name of WhosOn. It is meant to give you a real time snapshot of who is online and it can also issue some warnings for 404's and other stuff.

However, this program seems to use exactly the technique mentioned above. It indicates spider visits first by a list of known spider user agents, but then it has some heuristics which tries to detect previously unknown spiders. A dead giveaway is the request of robots.txt (even though I myself try to request it). Also not accepting cookies seems to be typical bot-behaviour.

[cfx211]

Refusing cookies is another tell tale sign of robots.

[Eugenios]

That one is good, but is there any way to check if a robot is refusing cookies other than using cookies? I suppose not.

Do site downloaders (like teleportpro) refuse cookies?

Enrique

[DanA]

Site downloaders accept cookies.