Inktomi spidering non-paying websites

[troels nybo nielsen]

It seems that Slurp over the last few days has begun deepcrawling my websites. I have certainly never paid anything for that.

New politics from the new owners? I have been visited before but always with only one hit.

[Go60Guy]

I don't know if this is the right place to get some clarification on this. But, I've heard that if you pay for inclusion for say, your home page, and not for your subpages, Inktomi will penalize your non-paid pages and bury them in remote, virtually inaccessable places.

Somehow, that doesn't sound right, but its a theory in circulation. Any substance to it?

[Callanish]

I use paid submission to get new promising pages spidered quickly.

As far as I can see when the new pages are picked up and included it has no effect on other freely included pages (from the same domain) at all - they stay in, and are just as visible as before.

[redzone]

One of the biggest changes in Inktomi's "free" crawling habits, is they now seem to "deep crawl" links they find on URL's they are crawling.

This is a big improvement from 1998-2001, when you had to submit "every" URL to Ink., if you wanted to insure that they were indexed.

We noticed a huge update around 1/10/2003, and Inktomi has been spidering hard since... :)

[HenryUK]

No payment, no submission.

I had to wait longer than I did with Google and FAST to get spidered and indexed but slurp has been a constant visitor, gradually hoovering up all of the content.

MSN referrals still below Google but show steady growth, now accounting for 25% of se referrals.

I don't know what I did right - I was focusing on getting indexed on Google and it just happened, albeit 4-6 weeks later than on Google/FAST.

I heard the rumours too and won't be paying for inclusion - too much to lose now!

[darex]

I would also love to know if the rumours are true that if you do a PFI for ink then it can be counter productive because pages from your site that are already in the index can be dropped.

It doesn't make sense for this to happen becuase if this was true and people knew about it then there goes ink's business model. However just because it doesn't make sense doesn't mean it isn't true!

A second (really newby question). how do you check your rankings in inktomi?

[HenryUK]

[search.positiontech.com...]

is a pure ink search.

[jbauder]

darex

A second (really newby question). how do you check your rankings in inktomi?

for the most part inktomi's results currently get maximum exposure on msn.com and hotbot.com I would suggest checking your rankings at those 2 spots,

you can visit inktomi for a list of their search partner sites

[makemetop]

>pages from your site that are already in the index can be dropped...

Some people say they have seen this, I tend to feel that the pages were dropped for other reasons. My own site has 2 paid for pages - yet all of the pages are indexed fine - and always have been.