Search Engine Spider and User Agent Identification Forum

I need to go back through some logs myself, I'm sure I have some of those in logs (although a couple of those Lyco's ones definately look new).

I'm thinking about changing the way I display the ip data on the ip list. I don't want to go to separate pages, but that page is becoming so popular and generating 5-10 emails a day, that I need to do something.

People are confused by the separate pages for known spider IP's and the followup domain/box ip lists. I tried to explain it better but I've only added to the confusion. I guess there is something about that page I'm not getting myself.

People seem to think those followup pages are all spiders. I get a couple emails a day, "I've been on the net x years and I've never seen a spider run from xyz host in my life". Or I get one to two a day, "why don't you have xyz ip on the spiders page?".

I still can't figure out what I should do there to clarify the situation for people. I've been trying to direct them here with questions.

One way of explaining the two different kinds of lists, could maybe be to say that the Spider IP list, is a list of known IP's from which a specific spider operates.
ie. 204.123.9.76 is Scooter/2.0+G.R.A.B+V1.1.0. Scooter is the nickname of The Alta Vista SPIDER.

The AltaVista related IP list (not the scooter spider ip list) is a list of IP's known to owned by AltaVista, and therefore you should keep an extra eye on these as they can be used if AV checks for cloaking, or they can switch IP,s so that Scooter might (very little chance imho) operate from one of these.

I use only the IP list in my cloaking.