6 bots left...

What's the safest identification method?

[menyak]

From what I've read in other threads, there are 6 core-algo-search engines left right now:

google
inktomi
AV/FAST
wisenut
teoma
openfind

What's the safest and most efficient way to identify their spiders when you search for a substring of $HTTP_SERVER_VARS['HTTP_USER_AGENT'] in php?

"googlebot" and "[e-mail]" will obviously work well for the first two. What about the other four?

[Jocelyn]

I advise you to use the ranges of well-known ip adresses of those spiders. It is a much safer way to identify them, as everyone can easily fake a user-agent.

[menyak]

I would basically just need it to omit session IDs to make the crawling easier, so the only benefit of faking the user agent would probably be that you get logged out all of the time, but I wouldn't see a security concern. :)

[menyak]

How about using...

"teomaagent"
"[e-mail]"
"WISEnutbot.com"
"[e-mail]"

Would that work? My concerns are that maybe one of the strings isn't up-to-date or is bound to change in due time. Would ALL the bots of the respective search engines contain these strings? Also, I wouldn't want to use anything that could also be part of a regular UA string, hence leaving a "normal" user without a proper session.

Thanks!

[Craig_F]

I need info on this too. What is the best way to identify all major spiders? I need to be as accurate as possible.

[menyak]

Is anyone able to help with this? It would really be MUCH appreciated! Thanks again.

[StopSpam]

I need info on this too. What is the best way to identify all major spiders? I need to be as accurate as possible.

===

The best way is indeed to check there IP adress ...
i got a tool for macintosh computer .. i am sure there
is something simular for PC ...

i type in the Ip adress of a robot ...

see sample: 64.68.82.46

my log says its from google. but to be 100% sure
i run it in this program it tells me this:

ip is from: crawler11.googlebot.com.

now i know for sure its google

[fiestagirl]

How about using [robotstxt.org?...]
HTTP User-Agent is listed, as is exclusion tag.