Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
Does anyone know if this is true... The reason Iím asking is that I have several internal pages that Iím optimizing and want to have indexed, but I donít want to pay for each one separately.
Or does Inktomi put a restriction on the spider to only crawl the paid URL?
Still, we have observed their behavior and I can answer your question.
1. Inktomi has made a dramatic change in their crawling behavior in the last couple of months. So, what I say here is based on their new pattern. You can find other threads in this forum that talk about a lot of their past behavior.
2. If you pay for a page, they spider it every 48 hours as they say.
3. If you pay for a non-home page on the site, inktomi will automatically include your home page in its crawl every 48 hours. Note this: If you want paid inclusion for your home page and another page on the site, you need only pay for the other page. The home page will be included automatically without paying. You are always throwing away good money to do PFI with Inktomi for a home page.
4. Inktomi runs two sets of spiders. The spiders that serve the paid inclusion program just spider the pages you pay for and do NOT follow links from these pages to crawl other pages.
5. Inktomi has another set of spiders that crawl the entire web, in the normal way, following links it finds to discover new pages. These are the spiders that have been recently re-invigorated after about a year of inactivity because Inktomi finally figured out that their database had become irrelevant and non-competitive as a source of information and had devolved into just a list of essentially paid advertisers.
6. Other threads in this forum discuss places where you can submit for free to get crawled by the inktomi free spider. Inktomi has attempted to close most of those avenues to free submission, but some still exist and it is still an effective way to get indexed for free.
7. Now to the heart of your question: What happens when you have a site with some pages on PFI and other pages that you haven't paid for but would like to see indexed? Your PFI pages will NOT help your non-pages get spidered and indexed. However (contrary to past behavior by Inktomi), if the free spider finds your non-paid pages through its normal crawling activity, those pages will be indexed.
The home page is the one and only page that will automatically be indexed by the "paid" spider. If you think about, there are many reasons why it makes sense for inktomi to do this.
It is not a contradiction with the rest of my point. The home page is not included because you link to it from a paid page. It is included because it is the home page of your site.