homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

New SE IP's

 12:00 am on Oct 2, 1999 (gmt 0)

Hi Brett

Most of my SE IP's came from you - thanks.

Today I've been going through 376 Mb logfiles (September) and I came up with some new ones.

Should this Info be for lurkers too, or can I join the secret mailinglist ;)

Well, here is something to start from, It's not on the current (28.09.99) searchengineworld spider ip list

All: Slurp.so/1.0

**Lycos** (T-Rex)
209's are Lycos_Spider_(T-Rex)

Both: Scooter/2.0 G.R.A.B. V1.1.0


Most of them were expected, and now they are confirmed.




 9:39 am on Oct 2, 1999 (gmt 0)

Oh, nice list.

I need to go back through some logs myself, I'm sure I have some of those in logs (although a couple of those Lyco's ones definately look new).

I'm thinking about changing the way I display the ip data on the ip list. I don't want to go to separate pages, but that page is becoming so popular and generating 5-10 emails a day, that I need to do something.

People are confused by the separate pages for known spider IP's and the followup domain/box ip lists. I tried to explain it better but I've only added to the confusion. I guess there is something about that page I'm not getting myself.

People seem to think those followup pages are all spiders. I get a couple emails a day, "I've been on the net x years and I've never seen a spider run from xyz host in my life". Or I get one to two a day, "why don't you have xyz ip on the spiders page?".

I still can't figure out what I should do there to clarify the situation for people. I've been trying to direct them here with questions.

Anyway, appreciate the confirms - next update I'll add them to the known spiders page.


 2:04 pm on Oct 4, 1999 (gmt 0)

Just updated the list. Thanks again - I did not have all the boxes from that one Lycos block at all (209.67.229.).


 12:17 pm on Oct 8, 1999 (gmt 0)

Hi Brett

One way of explaining the two different kinds of lists, could maybe be to say that the Spider IP list, is a list of known IP's from which a specific spider operates.
ie. is Scooter/2.0+G.R.A.B+V1.1.0. Scooter is the nickname of The Alta Vista SPIDER.

The AltaVista related IP list (not the scooter spider ip list) is a list of IP's known to owned by AltaVista, and therefore you should keep an extra eye on these as they can be used if AV checks for cloaking, or they can switch IP,s so that Scooter might (very little chance imho) operate from one of these.

I use only the IP list in my cloaking.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved