Forum Moderators: open
I get plenty of Slurp visits, around 5000-6000 per month. Yet only 300 out of my 30,000 or so pages are indexed. I have links to me from all over the web which why all the other search engines have included the majority of my pages in their indexes. Over the weekend I delved deep into the links pointing to me on the Yahoo index. I contacted several of the webmasters and asked if they were using paid inclusion for Inktomi? Most of these sites were commercial and most of them said yes.
My theory is this (here's hoping this one doesn't get 'edited' before you lot see it like so many of my theories do) if you are linked to by a site that is using Paid For Inclusion to be in the Inktomi index then the free Slurp crawler does not index your site. To be considered relevant and qualify for free inclusion you have to be linked to be a site that has also been listed for free.
I understand this motive, this means that webmasters who submit just their main URL via Paid Inclusion will not have the rest of their site indexed just from that link. (This happens frequently.) Whereas other webmasters state that submitting just a few of their site's pages via Paid Inclusion has gotten the rest of their site crawled and indexed for free. I would like the webmasters that have seen this to verify that they had no other links to them from free included sites in the Inktomi index.
Another reason for this is that the 'unethical' (in the UK we call them 'dodgy') webmasters would simply pay for inclusion of a links page that links to all his other sites and get them crawled for free.
This may all seem like greed and Yahoo/Inktomi trying to force every penny out of webmasters for their site. But I can see the positive side of how this eliminates a massive potential for spamming.
Can anyone help me confirm or disprove this theory?
added --> I can't believe I used the word 'gotten'. What a waste my education was ...
