Welcome to WebmasterWorld Guest from 23.20.230.24

Forum Moderators: Ocean10000 & incrediBILL

Message Too Old, No Replies

Ink Spider Business

Ink Spider Business

     
2:23 am on Jun 22, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Does anyone happen to know what each Inktomi spider is doing on it's visit? I know for example J6000 visits a day or two before getting indexed. And J101 through J109 seem to come by and may be partly to blame for pages that are dropped.

SI3000 and J4000+or- all seem to be more interested in home pages and/or index pages

So, does anyone know what the different INK spiders are doing on their visits???

Any thoughts
Steve

3:30 pm on Jun 22, 2000 (gmt 0)

10+ Year Member



That's a great question! Out of all the INK spiders, I've noticed them move in 3 different ways.
(1) One just hits the root page of that domain. And doesn't seem to crawl.
(2) One hits any other page from that domain other than the index page. This one doesn't seem to crawl either.
(3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.

That's what I have observed so far. I'm not sure how useful this is but if anyone finds the rest of the puzzle pieces, please let me know.

2:42 pm on Jun 23, 2000 (gmt 0)

10+ Year Member



(>>3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.<<

And which one would that be, pray tell? :)

2:55 pm on Jun 23, 2000 (gmt 0)

10+ Year Member



I forgot to add this to the previous post and couldn't edit it for some reason <shrug> anyway...

Any input would be most appreciated.

These are some of the ink spiders I've had on my server yesterday (this is across 5 sites):

Name: si520.inktomi.com = 1
Name: j100.inktomi.com = 1
Name: j5000.inktomisearch.com = 1
Name: si3000.inktomi.com = 1
Name: si3003.inktomi.com = 1
Name: si4001.inktomi.com = 1
Name: si4002.inktomi.com = 1
Name: j5004.inktomisearch.com = 1
Name: j5006.inktomi.com = 1
Name: wm3021.inktomi.com = 1
Name: q5100.inktomi.com = 1
Name: y1501.inktomi.com = 1

3:19 pm on Jun 23, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Never had INK do a full crawl. I hear rumors that it happens. The only ink spider I'm real sure of is J6000. When he shows up you're about to be indexed. But, he only looks at what's submitted to him. If you just send one page he only looks at one page. he doesn't take bait...

That's one of the reason I think INK doesn't care how many pages you submit in a day (but that could always change).

Steve

8:42 pm on Jun 25, 2000 (gmt 0)

10+ Year Member



Can anyone tell me what this is about? Not inktomi spiders but I didn't submit anything to AV.
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3

All the same day - all the same site, and again, I didn't submit it.

3:15 am on Jun 26, 2000 (gmt 0)

10+ Year Member



maybe someone else submitted for you, are they pages with links to other sites on them?

I regularly get people submitting pages that have links to their (I presume) site on them ...

9:59 am on Jun 26, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



Steve, have you been watching your logs close? What ink does, is a slow full crawl. They'll pull a page here, or a page there just real slow over time. The more pages you have, longer it takes.
5:08 pm on Jun 26, 2000 (gmt 0)

10+ Year Member



I've actually seen 2 INK spiders do a pretty good instant crawl.

UA: Mozilla/4.72 [en] (X11; U; NetBSD 1.4.2 i386; Nav)

UA: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)

4:27 pm on Jun 27, 2000 (gmt 0)

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



They are doing that as I write this on another one of our sites. (see the post in the ink msg board).
4:47 pm on Jun 27, 2000 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Kamikaze,

I saw the same unidentified agent with an Exodus communications IP crawl about a day and a half before the first hit. I have not been seeing identified Ink spiders in awhile on sites that have gotten listed. The only identified spider has been the dreaded Slurp/si on sites not being indexed.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month