Inktomi bot's deep crawl?

inktomisearch, slurp, crawl

[ircgeeks]

Ok is this normal? I have never seen deep crawl from slurp like this. It has been going on for two hour’s now they just keep coming... I am not complaining just surprised

66.196.72.68 j3158.inktomisearch.com Commercial 23 min : 25 sec
66.196.72.95 j3185.inktomisearch.com Commercial 27 min : 31 sec
66.196.72.71 j3161.inktomisearch.com Commercial 0 min : 18 sec
66.196.72.39 j3129.inktomisearch.com Commercial 0 min : 57 sec
66.196.72.11 j3101.inktomisearch.com Commercial 2 min : 49 sec
66.196.72.72 j3162.inktomisearch.com Commercial 16 min : 33 sec
66.196.72.81 j3171.inktomisearch.com Commercial 15 min : 35 sec
66.196.72.47 j3137.inktomisearch.com Commercial 17 min : 38 sec
66.196.72.62 j3152.inktomisearch.com Commercial 23 min : 48 sec
66.196.72.46 j3136.inktomisearch.com Commercial 25 min : 46 sec
66.196.72.40 j3130.inktomisearch.com Commercial 22 min : 34 sec
66.196.72.52 j3142.inktomisearch.com Commercial 7 min : 41 sec
66.196.72.60 j3150.inktomisearch.com Commercial 2 min : 50 sec
66.196.72.51 j3141.inktomisearch.com Commercial 12 min : 2 sec
66.196.72.35 j3125.inktomisearch.com Commercial 26 min : 39 sec
66.196.72.49 j3139.inktomisearch.com Commercial 6 min : 2 sec

[pendanticist]

si3005.inktomisearch.com - - [12/Apr/2003:05:00:49 -0700] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (Slurp/si; [e-mail]; http//www.inktomi.com/slurp.html)"
si3007.inktomisearch.com - - [12/Apr/2003:05:03:53 -0700] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (Slurp/si; [e-mail]; http//www.inktomi.com/slurp.html)"
j3122.inktomisearch.com - - [12/Apr/2003:05:04:52 -0700] "GET /Libraries.html HTTP/1.0" 200 14424 "-" "Mozilla/5.0 (Slurp/cat; [e-mail]; http//www.inktomi.com/slurp.html)"
j3122.inktomisearch.com - - [12/Apr/2003:05:05:32 -0700] "GET /Statistics.html HTTP/1.0" 200 22657 "-" "Mozilla/5.0 (Slurp/cat; [e-mail]; http//www.inktomi.com/slurp.html)"

I can't say too much about years past, but it's been like this long enough that to me, it is normal.

Pendanticist.

[flexsez]

This deep crawl behavior seems unusual to me too.

Within the past month my interior pages are showing up in MSN.

Part of the problem with MSN is that they keep changing their minds on what direction they are headed. Their serps are all over the map on a day to day basis.

When do the deep crawl results seem to show up in the serps?

[pendanticist]

>When do the deep crawl results seem to show up in the serps?

I don't concern myself w/serps, ergo I don't know how to answer your question. Sorry.

Pendanticist.

[coconutz]

>>When do the deep crawl results seem to show up in the serps?

Not sure when or if all of the pages crawled are added to the index, but Inktomi says the regular index is updated once every 2 weeks.

How often is Ink updating now? [webmasterworld.com]

[futureX]

i've had the same on and off for the past few weeks...

At one point inktomi was sending a new spider every 20 seconds which continued for around 12 hours (9/10 times the same page)... Looks like every fortnight inktomi comes and hamers my site... Although i have yet to see the pages it has visited in the SERPs

btw does anyone know a command on any of the inktomi sites that is equal to google's allinurl:?