Can you get Inktomi Slurp to index new pages?

It takes the old and ignores the new

[misosoph]

Every day Inktomi's slurp robot takes pages from my site -- but they are always the pages that it has already indexed. Even though there are links to my new pages on the old pages, slurp ignores the new pages. This has been going on for about two months now. (This isn't really a "how to" question; it's more: what's wrong with this robot? It's as if it were caught in a loop.)

[bobriggs]

I know I'm playing with fire, but Slurp keeps coming along and asking for pages that I have removed. So I decided to use mod_alias in Apache tonight to redirect it to similar NEW pages.

I've noticed that INK will show the old URL but index the content on the REDIRECTED page. After a while (AND, it is a long while), it will remove the old url in favor of the new.

If anyone can give me a warning signal about what I'm attempting, let me know. I'm in the same boat as misosoph, with old pages outdated, the new/updated pages with new urls need to be in.

[MarkHutch]

Let me know if that works. Ink does the same thing on our site. It keeps looking for pages that have been gone for years.

[keyplyr]

It's most likely following links from other sites, outdated as they may be. If you could repair those old links by contacting the referrals that send traffic to your redirects, that might reduce some of the problem.

[MarkHutch]

>>If you could repair those old links by contacting the referrals that send traffic to your redirects<<

That's what I thought too, but it's not. Some of the pages it's trying to get are not indexed anywhere else and the strange thing is that sometimes Ink tries to spider a page that's no longer there and returns a 304, "page not modified" result. It kind of hard to have a not modified result when the page has been gone for years.

[bobriggs]

and returns a 304, "page not modified" result. It kind of hard to have a not modified result when the page has been gone for years..

Well are you sure the page isn't there? Or is it just delinked from your site? The pages I'm going to attempt are 404 - Ink has been receiving 404 errors for months now, and I'm just going to give it a new address.

If you have a page that isn't on your server returning 304 there's something wrong with the server.

[rzfree]

Only guess but I can see corelation of the number of inbound links from other sites to number of pages that Ink. "wants" to have in their DB. On one site, once recip. campain became stale, no new pages added for 8 months.
rz