How deep does paid spidering go?

Why would I need to pay for multiple urls in the same site?

[AkanDian rain]

Let me start by saying I LOVE paid spider. I was never very patient to begin with. However, I’ve always wondered about how deep the spider goes for the money. Is seems reasonable to me that if you paid for your index.html to be spidered, that would lead to the natural spidering of the entire Web site.

Does anyone know if this is true... The reason I’m asking is that I have several internal pages that I’m optimizing and want to have indexed, but I don’t want to pay for each one separately.

[Inktomi]

Inktomi's paid inclusion program is a pay-per-URL program, so you www.yoursite.com/, www.yoursite.com/foo/ and www.yoursite.com/bar/ are all separate URLs and are priced separately. The Search Submit price is $39/year for the first URL and $25/year for each subsequent URL. If you have more than 1,000 URLs, you can participate in the Index Connect program, which is a pay-for-performance rather than pay-per-URL program.

Inktomi.

[AkanDian rain]

Thank you Inktomi for your reply. That was pretty much the same information provided by the Web site. And perhaps you are not at liberty to discuss the theory behind this question. But it seems to me that paying to have the root URL of your Web site indexed by Inktomi would “stimulate” the natural spidering of the rest of the Web site at the same time.

Or does Inktomi put a restriction on the spider to only crawl the paid URL?

[Jack_Straw]

I doubt that Inktomi could be less forthcoming with his very "political" answer. I fear that their PFI business model again puts Inktomi in a position of trying to to say two contradictory things at once. On the one hand, they must give people reason to pay to be included in their database. And on the other hand, they much index as must of the web as possible to compete with Google and Fast.

Still, we have observed their behavior and I can answer your question.

1. Inktomi has made a dramatic change in their crawling behavior in the last couple of months. So, what I say here is based on their new pattern. You can find other threads in this forum that talk about a lot of their past behavior.

2. If you pay for a page, they spider it every 48 hours as they say.

3. If you pay for a non-home page on the site, inktomi will automatically include your home page in its crawl every 48 hours. Note this: If you want paid inclusion for your home page and another page on the site, you need only pay for the other page. The home page will be included automatically without paying. You are always throwing away good money to do PFI with Inktomi for a home page.

4. Inktomi runs two sets of spiders. The spiders that serve the paid inclusion program just spider the pages you pay for and do NOT follow links from these pages to crawl other pages.

5. Inktomi has another set of spiders that crawl the entire web, in the normal way, following links it finds to discover new pages. These are the spiders that have been recently re-invigorated after about a year of inactivity because Inktomi finally figured out that their database had become irrelevant and non-competitive as a source of information and had devolved into just a list of essentially paid advertisers.

6. Other threads in this forum discuss places where you can submit for free to get crawled by the inktomi free spider. Inktomi has attempted to close most of those avenues to free submission, but some still exist and it is still an effective way to get indexed for free.

7. Now to the heart of your question: What happens when you have a site with some pages on PFI and other pages that you haven't paid for but would like to see indexed? Your PFI pages will NOT help your non-pages get spidered and indexed. However (contrary to past behavior by Inktomi), if the free spider finds your non-paid pages through its normal crawling activity, those pages will be indexed.

[web_india]

>>If you pay for a non-home page on the site, inktomi will automatically include your home page in its crawl every 48 hours.

Jack, Are you sure of this? Because you are yourself saying

"Your PFI pages will NOT help your non-pages get spidered and indexed."

What is that I am missing?

[Jack_Straw]

Yes, we have multiple confirmations that the free home page thing is real.

The home page is the one and only page that will automatically be indexed by the "paid" spider. If you think about, there are many reasons why it makes sense for inktomi to do this.

It is not a contradiction with the rest of my point. The home page is not included because you link to it from a paid page. It is included because it is the home page of your site.

[AkanDian rain]

Good Info Jack,

Do you know if that is true for the other paid spiders?

[Jack_Straw]

All that applies only to Inktomi.

[web_india]

Interesting find, Jack.

So is there any time gap when the home page will be included along with your PFI non-home page? Or, it will happen simultaneously?

[Jack_Straw]

Slurp does not begin crawling the home page right away after you have paid for inclusion of another page on the site. But, it begins crawling after a short time (it looks like a couple of weeks to a month from the site I just checked).