homepage Welcome to WebmasterWorld Guest from 54.167.179.48
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Ink Spider Business
Ink Spider Business
stcrim

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 79 posted 2:23 am on Jun 22, 2000 (gmt 0)

Does anyone happen to know what each Inktomi spider is doing on it's visit? I know for example J6000 visits a day or two before getting indexed. And J101 through J109 seem to come by and may be partly to blame for pages that are dropped.

SI3000 and J4000+or- all seem to be more interested in home pages and/or index pages

So, does anyone know what the different INK spiders are doing on their visits???

Any thoughts
Steve

 

Kamikaze

10+ Year Member



 
Msg#: 79 posted 3:30 pm on Jun 22, 2000 (gmt 0)

That's a great question! Out of all the INK spiders, I've noticed them move in 3 different ways.
(1) One just hits the root page of that domain. And doesn't seem to crawl.
(2) One hits any other page from that domain other than the index page. This one doesn't seem to crawl either.
(3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.

That's what I have observed so far. I'm not sure how useful this is but if anyone finds the rest of the puzzle pieces, please let me know.

jilly

10+ Year Member



 
Msg#: 79 posted 2:42 pm on Jun 23, 2000 (gmt 0)

(>>3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.<<

And which one would that be, pray tell? :)

jilly

10+ Year Member



 
Msg#: 79 posted 2:55 pm on Jun 23, 2000 (gmt 0)

I forgot to add this to the previous post and couldn't edit it for some reason <shrug> anyway...

Any input would be most appreciated.

These are some of the ink spiders I've had on my server yesterday (this is across 5 sites):

Name: si520.inktomi.com = 1
Name: j100.inktomi.com = 1
Name: j5000.inktomisearch.com = 1
Name: si3000.inktomi.com = 1
Name: si3003.inktomi.com = 1
Name: si4001.inktomi.com = 1
Name: si4002.inktomi.com = 1
Name: j5004.inktomisearch.com = 1
Name: j5006.inktomi.com = 1
Name: wm3021.inktomi.com = 1
Name: q5100.inktomi.com = 1
Name: y1501.inktomi.com = 1

stcrim

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 79 posted 3:19 pm on Jun 23, 2000 (gmt 0)

Never had INK do a full crawl. I hear rumors that it happens. The only ink spider I'm real sure of is J6000. When he shows up you're about to be indexed. But, he only looks at what's submitted to him. If you just send one page he only looks at one page. he doesn't take bait...

That's one of the reason I think INK doesn't care how many pages you submit in a day (but that could always change).

Steve

jilly

10+ Year Member



 
Msg#: 79 posted 8:42 pm on Jun 25, 2000 (gmt 0)

Can anyone tell me what this is about? Not inktomi spiders but I didn't submit anything to AV.
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3

All the same day - all the same site, and again, I didn't submit it.

xAir

10+ Year Member



 
Msg#: 79 posted 3:15 am on Jun 26, 2000 (gmt 0)

maybe someone else submitted for you, are they pages with links to other sites on them?

I regularly get people submitting pages that have links to their (I presume) site on them ...

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 79 posted 9:59 am on Jun 26, 2000 (gmt 0)

Steve, have you been watching your logs close? What ink does, is a slow full crawl. They'll pull a page here, or a page there just real slow over time. The more pages you have, longer it takes.

Kamikaze

10+ Year Member



 
Msg#: 79 posted 5:08 pm on Jun 26, 2000 (gmt 0)

I've actually seen 2 INK spiders do a pretty good instant crawl.

UA: Mozilla/4.72 [en] (X11; U; NetBSD 1.4.2 i386; Nav)

UA: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 79 posted 4:27 pm on Jun 27, 2000 (gmt 0)

They are doing that as I write this on another one of our sites. (see the post in the ink msg board).

JamesR

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 79 posted 4:47 pm on Jun 27, 2000 (gmt 0)

Kamikaze,

I saw the same unidentified agent with an Exodus communications IP crawl about a day and a half before the first hit. I have not been seeing identified Ink spiders in awhile on sites that have gotten listed. The only identified spider has been the dreaded Slurp/si on sites not being indexed.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved