Forum Moderators: phranque
It appears as if Google picked everything up just fine after the last "dance".
I have noticed a big difference in the Inktomi bot however. My old CMS used a /modules.php page for much of the content, the new one does not so I copied my new index page to the old modules.php location so anyone clicking on an old link would at least go to the main page.
The Inktomi bot seems to be validating links, and has been doing so every 10 minutes for probably 3 months now. All of the links to be verified are all the same old /modules.php that is actually the new index.php page, the bot has not yet found even one new page!?
I souldn't really complain, I have no idea how Inktomi picked me up and I am glad that I am there without paying anything. Just curious if I have done the right thing my putting the phantom page there, should I take it out so Inktomi will be forced to recrawl? or will the bot finally realize that all of these links are actually the same?
I think that might be part of the problem.. You have a real page there. Possibly also link popularity for that page. You could try a 404 catch-all page or there are some other permanent redirect codes that I am less familiar with.
As regards to Inktomi not finding the new page but Google did... maybe means the link popularity is less of a factor... maybe you did the paid inclussion a while back for Inktomi and they are still seeing a page.
Should you take it out? I think I would unless it was getting good traffic. And if it is, make it a good topical landing page.
PS - Thanks :)
Well, I am still getting some "real" hit's from the original links so I am somewhat hesitant on taking the page out completely, I could use a 404 but was thinking of an "update your bookmarks" page with a redirect. I don't know how bot's would handle this though, probably some would drop my listings since it would be just a link forwarding to a different domain.
Also, I think that Inktomi would find all the new pages once it actually crawled the new site. It has been since early January, more than 2 months, since I have changes the pages. This seems like a long time inbetween crawls.
I do like the idea of making a new page and putting it there, I could put a few links to lead the bot's over to the new pages. hmmm
Thanks for the idea!
BTW - Welcome to WebmasterWorld. ;-)
Slurp's behavior was strange and stubborn when it was under Inktomi. Now, I think a lot of people are watching to see if it does the same things under Yahoo.
I am currently experimenting with error codes 301 (permanent redirect), 302 (temporary redirect), 404 (not found), and 410 (permanently removed).
One thing is for sure, even when you figure out the right response code for your situation, you have to wait a LONG TIME to see it take effect. Even the 410 permanently removed tag will not make Slurp lay off that URL for a good long while.
The real objective is to take your popular pages, permanently redirect them to the new pages, let Slurp discover the new pages (through that redirect or else-wise), and then when you see Slurp finally visit the new page, remove the old page and generate a 410 permanent removal code. That's the clean and proper way to do it, and the way Slurp SHOULD abide by.
