Welcome to WebmasterWorld Guest from 184.72.148.222

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Ink Spider Business

Ink Spider Business

     
2:23 am on Jun 22, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 4, 2000
posts:1324
votes: 0


Does anyone happen to know what each Inktomi spider is doing on it's visit? I know for example J6000 visits a day or two before getting indexed. And J101 through J109 seem to come by and may be partly to blame for pages that are dropped.

SI3000 and J4000+or- all seem to be more interested in home pages and/or index pages

So, does anyone know what the different INK spiders are doing on their visits???

Any thoughts
Steve

3:30 pm on June 22, 2000 (gmt 0)

Junior Member

10+ Year Member

joined:June 20, 2000
posts:151
votes: 0


That's a great question! Out of all the INK spiders, I've noticed them move in 3 different ways.
(1) One just hits the root page of that domain. And doesn't seem to crawl.
(2) One hits any other page from that domain other than the index page. This one doesn't seem to crawl either.
(3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.

That's what I have observed so far. I'm not sure how useful this is but if anyone finds the rest of the puzzle pieces, please let me know.

jilly

2:42 pm on June 23, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


(>>3) Finally, they have a certain spider for crawling purpose. I think this is the type of spider you want most.<<

And which one would that be, pray tell? :)

jilly

2:55 pm on June 23, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


I forgot to add this to the previous post and couldn't edit it for some reason <shrug> anyway...

Any input would be most appreciated.

These are some of the ink spiders I've had on my server yesterday (this is across 5 sites):

Name: si520.inktomi.com = 1
Name: j100.inktomi.com = 1
Name: j5000.inktomisearch.com = 1
Name: si3000.inktomi.com = 1
Name: si3003.inktomi.com = 1
Name: si4001.inktomi.com = 1
Name: si4002.inktomi.com = 1
Name: j5004.inktomisearch.com = 1
Name: j5006.inktomi.com = 1
Name: wm3021.inktomi.com = 1
Name: q5100.inktomi.com = 1
Name: y1501.inktomi.com = 1

3:19 pm on June 23, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 4, 2000
posts:1324
votes: 0


Never had INK do a full crawl. I hear rumors that it happens. The only ink spider I'm real sure of is J6000. When he shows up you're about to be indexed. But, he only looks at what's submitted to him. If you just send one page he only looks at one page. he doesn't take bait...

That's one of the reason I think INK doesn't care how many pages you submit in a day (but that could always change).

Steve

jilly

8:42 pm on June 25, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


Can anyone tell me what this is about? Not inktomi spiders but I didn't submit anything to AV.
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3
Name: add-url.pa.alta-vista.net = 3

All the same day - all the same site, and again, I didn't submit it.

xAir

3:15 am on June 26, 2000 (gmt 0)

Inactive Member
Account Expired

 
 


maybe someone else submitted for you, are they pages with links to other sites on them?

I regularly get people submitting pages that have links to their (I presume) site on them ...

9:59 am on June 26, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38057
votes: 12


Steve, have you been watching your logs close? What ink does, is a slow full crawl. They'll pull a page here, or a page there just real slow over time. The more pages you have, longer it takes.
5:08 pm on June 26, 2000 (gmt 0)

Junior Member

10+ Year Member

joined:June 20, 2000
posts:151
votes: 0


I've actually seen 2 INK spiders do a pretty good instant crawl.

UA: Mozilla/4.72 [en] (X11; U; NetBSD 1.4.2 i386; Nav)

UA: Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)

4:27 pm on June 27, 2000 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38057
votes: 12


They are doing that as I write this on another one of our sites. (see the post in the ink msg board).
4:47 pm on June 27, 2000 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 23, 2000
posts:1277
votes: 0


Kamikaze,

I saw the same unidentified agent with an Exodus communications IP crawl about a day and a half before the first hit. I have not been seeing identified Ink spiders in awhile on sites that have gotten listed. The only identified spider has been the dreaded Slurp/si on sites not being indexed.

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members