Welcome to WebmasterWorld Guest from 107.22.14.254

Forum Moderators: open

Message Too Old, No Replies

Inktomi bot's deep crawl?

inktomisearch, slurp, crawl

     
7:52 pm on Apr 11, 2003 (gmt 0)

10+ Year Member



Ok is this normal? I have never seen deep crawl from slurp like this. It has been going on for two hourís now they just keep coming... I am not complaining just surprised


66.196.72.68 j3158.inktomisearch.com Commercial 23 min : 25 sec
66.196.72.95 j3185.inktomisearch.com Commercial 27 min : 31 sec
66.196.72.71 j3161.inktomisearch.com Commercial 0 min : 18 sec
66.196.72.39 j3129.inktomisearch.com Commercial 0 min : 57 sec
66.196.72.11 j3101.inktomisearch.com Commercial 2 min : 49 sec
66.196.72.72 j3162.inktomisearch.com Commercial 16 min : 33 sec
66.196.72.81 j3171.inktomisearch.com Commercial 15 min : 35 sec
66.196.72.47 j3137.inktomisearch.com Commercial 17 min : 38 sec
66.196.72.62 j3152.inktomisearch.com Commercial 23 min : 48 sec
66.196.72.46 j3136.inktomisearch.com Commercial 25 min : 46 sec
66.196.72.40 j3130.inktomisearch.com Commercial 22 min : 34 sec
66.196.72.52 j3142.inktomisearch.com Commercial 7 min : 41 sec
66.196.72.60 j3150.inktomisearch.com Commercial 2 min : 50 sec
66.196.72.51 j3141.inktomisearch.com Commercial 12 min : 2 sec
66.196.72.35 j3125.inktomisearch.com Commercial 26 min : 39 sec
66.196.72.49 j3139.inktomisearch.com Commercial 6 min : 2 sec
3:27 pm on Apr 12, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



si3005.inktomisearch.com - - [12/Apr/2003:05:00:49 -0700] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http//www.inktomi.com/slurp.html)"
si3007.inktomisearch.com - - [12/Apr/2003:05:03:53 -0700] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (Slurp/si; slurp@inktomi.com; http//www.inktomi.com/slurp.html)"
j3122.inktomisearch.com - - [12/Apr/2003:05:04:52 -0700] "GET /Libraries.html HTTP/1.0" 200 14424 "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http//www.inktomi.com/slurp.html)"
j3122.inktomisearch.com - - [12/Apr/2003:05:05:32 -0700] "GET /Statistics.html HTTP/1.0" 200 22657 "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http//www.inktomi.com/slurp.html)"

I can't say too much about years past, but it's been like this long enough that to me, it is normal.

Pendanticist.

4:50 am on Apr 13, 2003 (gmt 0)

10+ Year Member



This deep crawl behavior seems unusual to me too.

Within the past month my interior pages are showing up in MSN.

Part of the problem with MSN is that they keep changing their minds on what direction they are headed. Their serps are all over the map on a day to day basis.

When do the deep crawl results seem to show up in the serps?

10:23 pm on Apr 13, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>When do the deep crawl results seem to show up in the serps?

I don't concern myself w/serps, ergo I don't know how to answer your question. Sorry.

Pendanticist.

1:43 am on Apr 14, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



>>When do the deep crawl results seem to show up in the serps?

Not sure when or if all of the pages crawled are added to the index, but Inktomi says the regular index is updated once every 2 weeks.

How often is Ink updating now? [webmasterworld.com]

1:49 am on Apr 15, 2003 (gmt 0)

10+ Year Member



i've had the same on and off for the past few weeks...

At one point inktomi was sending a new spider every 20 seconds which continued for around 12 hours (9/10 times the same page)... Looks like every fortnight inktomi comes and hamers my site... Although i have yet to see the pages it has visited in the SERPs

btw does anyone know a command on any of the inktomi sites that is equal to google's allinurl:?

 

Featured Threads

Hot Threads This Week

Hot Threads This Month