Brain dead Inktomi spider

[badtzmaru]

Does anyone else find that the Inktomi spider is totally brain-dead? I have a dynamic site and Inktomi's spider is the only one that acts in a completely moronic manner. yj1016.inktomi.com spiders me probably at least once every 5 minutes, continuous, for like 4 months now... for example, in January it hit me 76,000 times and transferred 500MB of content! Thats ridiculous!

[olias]

Yeah, I like the way that it refuses to accept that pages are dead. I have some old pages with no links any longer that have been gone for over two years, but Slurp insists on returning every day hoping that maybe it wont find the same old 404, i am half tempted to 301 it to pages that are in more need of its attention

[badtzmaru]

Yeah, you're right there! I found a page it was trying to spider that hadn't been there for well over a year... when I noticed it, I just set up a redirect to my site index...

[allanp73]

Do you know how to get Inktomi to spider new sites and how to get it to spider deeply?
In the past I used Swisssearch and other se's powered by Inktomi now they are gone or their submit url pages are just affliates of Inktomi.
I know that DMOZ is one way in but find the submission process slow and my niche is even more difficult. So how does one get the Inktomi spider to visit (a dumb spider is better than no spider at all)?

[robotsdobetter]

Slurp has been indexing one of my pages that are 404s for over 3 years now. I own some other sites and they do the same thing their to, but Googlebot does to, but most of time it don't. I found that it's better to cloke your 404 then to have it redirect. Slurp also indexes the same page over and over the same day. It's been doing for about 2 months now. All I have to say is Google rules and Inktomi sucks!