Ink Spider Business

[stcrim]

Does anyone happen to know what each Inktomi spider is doing on it's visit? I know for example J6000 visits a day or two before getting indexed. And J101 through J109 seem to come by and may be partly to blame for pages that are dropped.

SI3000 and J4000+or- all seem to be more interested in home pages and/or index pages

So, does anyone know what the different INK spiders are doing on their visits???

Any thoughts
Steve

[Kamikaze]

That's a great question! Out of all the INK spiders, I've noticed them move in 3 different ways.
(1) One just hits the root page of that domain. And doesn't seem to crawl.
(2) One hits any other page from that domain other than the index page. This one doesn't seem to crawl either.
(3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.

That's what I have observed so far. I'm not sure how useful this is but if anyone finds the rest of the puzzle pieces, please let me know.

[jilly]

(>>3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.<<

And which one would that be, pray tell? :)

[jilly]

I forgot to add this to the previous post and couldn't edit it for some reason <shrug> anyway...

Any input would be most appreciated.

These are some of the ink spiders I've had on my server yesterday (this is across 5 sites):

Name: si520.inktomi.com = 1
Name: j100.inktomi.com = 1
Name: j5000.inktomisearch.com = 1
Name: si3000.inktomi.com = 1
Name: si3003.inktomi.com = 1
Name: si4001.inktomi.com = 1
Name: si4002.inktomi.com = 1
Name: j5004.inktomisearch.com = 1
Name: j5006.inktomi.com = 1
Name: wm3021.inktomi.com = 1
Name: q5100.inktomi.com = 1
Name: y1501.inktomi.com = 1

[stcrim]

Never had INK do a full crawl. I hear rumors that it happens. The only ink spider I'm real sure of is J6000. When he shows up you're about to be indexed. But, he only looks at what's submitted to him. If you just send one page he only looks at one page. he doesn't take bait...

That's one of the reason I think INK doesn't care how many pages you submit in a day (but that could always change).

Steve

[jilly]

Can anyone tell me what this is about? Not inktomi spiders but I didn't submit anything to AV.
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3

All the same day - all the same site, and again, I didn't submit it.

[xAir]

maybe someone else submitted for you, are they pages with links to other sites on them?

I regularly get people submitting pages that have links to their (I presume) site on them ...

[Brett_Tabke]

Steve, have you been watching your logs close? What ink does, is a slow full crawl. They'll pull a page here, or a page there just real slow over time. The more pages you have, longer it takes.

[Kamikaze]

I've actually seen 2 INK spiders do a pretty good instant crawl.

UA: Mozilla/4.72 [en] (X11; U; NetBSD 1.4.2 i386; Nav)

UA: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)

[Brett_Tabke]

They are doing that as I write this on another one of our sites. (see the post in the ink msg board).

[JamesR]

Kamikaze,

I saw the same unidentified agent with an Exodus communications IP crawl about a day and a half before the first hit. I have not been seeing identified Ink spiders in awhile on sites that have gotten listed. The only identified spider has been the dreaded Slurp/si on sites not being indexed.