More pages get spidered for free

Maybe it's better not to pay Inktomi

[dvduval]

I have one site where I pay for inclusion and another I don't pay for. The site that I don't pay for has many more pages included in Inktomi. Doesn't seem fair. Anyone else have this problem?

[fathom]

All sites will eventually be spidered, and for free. The paid for inclusion is for impatient people!

[dvduval]

Both sites have been around for a couple of years...so that's not an issue in this case.

[deejay]

ahh.. but the big question is, what does your robots tag say on the included pages?

There's been a post here recently noting that Ink's default robot tag interpretation is "follow, noindex" (which I've also read somewhere else in recent months)... implying that the only way to get included in Inktomi when spidered is to have a "follow, index" tag on the page.

You might be just the person to solve the mystery, dvduval.

[fathom]

Actually I should re-state my comment. Paid for inclusion pages generally receive better results in ranked position. So on one hand free gives more and on the other paid for gives better.

<added> deejay post is very true!</added>

[edited by: fathom at 12:40 am (utc) on July 21, 2002]

[dvduval]

I don't have a robots tag on either site. Are you saying I should put a specific tag on every page on both sites?

[fathom]

Yes, because Inktomi's default is no follow and therefore your site will never be indexed totally, unless paying through the nose of course.

Add to each page Header:

<META NAME="ROBOTS" CONTENT="index,follow">

[dvduval]

Yes, thank you.
Here is some supporting evidence:
[searchengineworld.com...]