Inktomi/MSN - Inclusion

Inktomi/MSN - Inclusion Help!

[Catnip]

"Inktomi® Search Submit™ is the best way to submit up to 1,000 pages for search engine inclusion." Is this the only way to get multiple pages on MSN? I already have our main page listed on MSN, however we have around 1500. 1st URL $39, URLs 2-1000 $25 each. So if we submit 500 pages it will cost $12,514.00 for a year correct? Any other method of getting in?

Thanks,

Catnip

[Mohamed_E]

> Any other method of getting in?

Inktomi seems to be spidering the entire web for free. At some time in August they picked up my entire (small) site, I have never paid them a penny.

[Catnip]

This data is from WebTrends Enterprise Reporting Server 3.5c for the month of November.

Spider Hits
1. Googlebot 41,735

3. TurnitinBot 7,058

4. Openfind data gatherer, Openbot 493

5. FAST-WebCrawler 180

6. Mozilla/5.0 (Slurp/cat; [e-mail]; [inktomi.com...] 166

7. Mozilla/5.0 (Slurp/si; [e-mail]; [inktomi.com...] 37

8. scooter 22

9. http: 11

10. BaiDuSpider 11

11. Mozilla/3.0 (Slurp/si; [e-mail]; [inktomi.com...] 9

12. Mozilla/3.0 (Slurp.so/Goo; [e-mail]; [inktomi.com...] 7

13. Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; MSIECrawler) 4

14. URL Spider Pro 3

15. Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; MSIECrawler) 2

16. Steeler 2

17. ExactSeek Crawler 2

18. Mozilla/4.0 (compatible; MSIE 5.5; Windows 98; Win 9x 4.90; YComp 5.0.2.4; MSIECrawler) 2

19. Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Q312461; Hotbar 4.1.8.0; MSIECrawler) 1

20. Robot: NutchCrawler, Owner: [e-mail] 1

It just seems to check a few pages and that is it? What could the problem be? And is this a poor crawl of search engines for a month? Does MSN and Altavista crawl once a month? I don't see a lot of the big engines every month should I?

[mayor]

>> It just seems to check a few pages and that is it? What could the problem be?

I've been watching Ink closely since early September and have seen similar dismal spidering activity. At the same time, I had one free inclusion site and one paid inclusion site dumped ("dumped" means the free sites pages cannot be found at all in their index and PFI pages are absolutely buried at the bottom of the barrel).

I also have a couple new sites put on the Web in September and Google, Alta Vista, Fast and others have picked them up but not Inktomi.

It looks to me like they're unsuccessfully fighting spam for their lives (because the PFI site they dumped had been in their index for over a year, and is not spam as far as I and all the other search engines are concerned) so they are having to crank up their spam cleansing so radically as to cut off their noses to spite their face. Still, it's easy to find tons of spam in their index.

It could also be that they can't afford the cost of spidering the entire web anymore.

And maybe they can no longer afford the cost of enough skilled technical manpower to maintain their systems and algorithms properly. Being from the programming world myself, I know when the chips are down one of the first things management wants to do is cut expensive technical resources and stop thoroughly testing code before it goes out the door, usually with disasterous consequences.

My window for analysis is very small and based on a few small websites of less than 50 pages, but I'm trying to contribute my best two cents worth toward understanding the problem.

What's Inktomi "guy's" take on it all? Some real facts from the expert would help dispell any erroneous conclusions based on limited information.

[Catnip]

Anyone else have some insight?

[makemetop]

PFI is the way to fly :)

[onlineleben]

>PFI is the way to fly<
But too expensive for hobby sites or .orgs

Had a similar problem with slurp not spidering more than my index page. I included <META NAME="ROBOTS" CONTENT="index,follow"> and now it also spiders the other pages.
Only problem remaining is that it comes once per day, picks just 1 page and leaves. Any cure for this?

[jimh009]

I'm having the same problem. In September and October, Inktomi use to come around once a day. It took in the index page and left, although I've never been able to find the index page in a search. I added a robots.txt file and a "Index, Follow" in the meta commands hoping it would help. It didn't.

Now Inktomi only comes around once a week or so, compared to almost their daily visits before. And as always, they take in the index page and leave.

I have pretty much given up on getting into Ink, since the free route doesn't seem to work and I have a 1500 page hobby web site that I can hardly afford to pay for inclusion.

At least AV and FAST have gotten at least part of my site over the past month. Still no comparison to Google, though, which faithfully "slurps" in my entire site every month.