Welcome to WebmasterWorld Guest from 126.96.36.199 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Become a Pro Member
Inktomi bot's deep crawl? inktomisearch, slurp, crawl ircgeeks
Ok is this normal? I have never seen deep crawl from slurp like this. It has been going on for two hourís now they just keep coming... I am not complaining just surprised
188.8.131.52 j3158.inktomisearch.com Commercial 23 min : 25 sec 184.108.40.206 j3185.inktomisearch.com Commercial 27 min : 31 sec 220.127.116.11 j3161.inktomisearch.com Commercial 0 min : 18 sec 18.104.22.168 j3129.inktomisearch.com Commercial 0 min : 57 sec 22.214.171.124 j3101.inktomisearch.com Commercial 2 min : 49 sec 126.96.36.199 j3162.inktomisearch.com Commercial 16 min : 33 sec 188.8.131.52 j3171.inktomisearch.com Commercial 15 min : 35 sec 184.108.40.206 j3137.inktomisearch.com Commercial 17 min : 38 sec 220.127.116.11 j3152.inktomisearch.com Commercial 23 min : 48 sec 18.104.22.168 j3136.inktomisearch.com Commercial 25 min : 46 sec 22.214.171.124 j3130.inktomisearch.com Commercial 22 min : 34 sec 126.96.36.199 j3142.inktomisearch.com Commercial 7 min : 41 sec 188.8.131.52 j3150.inktomisearch.com Commercial 2 min : 50 sec 184.108.40.206 j3141.inktomisearch.com Commercial 12 min : 2 sec 220.127.116.11 j3125.inktomisearch.com Commercial 26 min : 39 sec 18.104.22.168 j3139.inktomisearch.com Commercial 6 min : 2 sec
si3005.inktomisearch.com - - [12/Apr/2003:05:00:49 -0700] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (Slurp/si; firstname.lastname@example.org; http//www.inktomi.com/slurp.html)" si3007.inktomisearch.com - - [12/Apr/2003:05:03:53 -0700] "GET /robots.txt HTTP/1.0" 200 220 "-" "Mozilla/5.0 (Slurp/si; email@example.com; http//www.inktomi.com/slurp.html)" j3122.inktomisearch.com - - [12/Apr/2003:05:04:52 -0700] "GET /Libraries.html HTTP/1.0" 200 14424 "-" "Mozilla/5.0 (Slurp/cat; firstname.lastname@example.org; http//www.inktomi.com/slurp.html)" j3122.inktomisearch.com - - [12/Apr/2003:05:05:32 -0700] "GET /Statistics.html HTTP/1.0" 200 22657 "-" "Mozilla/5.0 (Slurp/cat; email@example.com; http//www.inktomi.com/slurp.html)"
I can't say too much about years past, but it's been like this long enough that to me, it is normal.
This deep crawl behavior seems unusual to me too.
Within the past month my interior pages are showing up in MSN.
Part of the problem with MSN is that they keep changing their minds on what direction they are headed. Their serps are all over the map on a day to day basis.
When do the deep crawl results seem to show up in the serps?
>When do the deep crawl results seem to show up in the serps?
I don't concern myself w/serps, ergo I don't know how to answer your question. Sorry.
>>When do the deep crawl results seem to show up in the serps?
Not sure when or if all of the pages crawled are added to the index, but Inktomi says the regular index is updated once every 2 weeks.
How often is Ink updating now? [ webmasterworld.com] futureX
i've had the same on and off for the past few weeks...
At one point inktomi was sending a new spider every 20 seconds which continued for around 12 hours (9/10 times the same page)... Looks like every fortnight inktomi comes and hamers my site... Although i have yet to see the pages it has visited in the SERPs
btw does anyone know a command on any of the inktomi sites that is equal to google's allinurl:?