inktomi spiders me, why?

[helenp]

Hi,

I considering pay Inktomi to get include in their directory, especially knowing yahoo is going to use Inktomis database.

But what I donīt understand is why i often see inktomi spidering my site......... this spider: (Slurp/si; [e-mail]; http://www.inktomi.com/slurp.html)

I am not included in their database, and I am not appearing in msn either, my site is comercial, so I guess it wanīt be included for free........

[cbpayne]

Inktomi is not a pay for inclusion search engine. It is a crawler based search engine (like Google). If you have good links to your site, you will be in Inktomi.

ALl the payment to Inktomi does is guarantee that you get crawled every 48hrs

[helenp]

Yes, but I canīt find my domain on their database, nor in msn or hotbot........... I had good links for a year.
If so, why am I not included, does it take very, very long?
If so, that does mean as well that there are no need to pay Inktomi to make sure you will be in Yahoos websearch when they switch to Inktomi?

Thanks

[cbpayne]

Can you sticky mail me the URL?

[helenp]

just sent the sticky,
hope I did it correctly,
ever done before

[cbpayne]

You are not in the Inktomi index as far as I can see. There are a lot of sites that link to you that are in the Ink index --> maybe wait and see what happens. Not sure of reasons, but maybe others with more experience on this can comment.

At the end of the day we do not know as much about Inktomi as we do about Google (...well maybe not the new Google), but Google do not list some sites, so the same must be true of Inktomi.

Inktomi have a smaller index than Google, so they must index less (hopefully that will change).

[helenp]

Thanks
Trying to read the rawlog, because everytime I saw inktomi in my stats, (not to good) it always stated my robots.txt file.
As far as I can see, reading it, itīs dificult without a program, it stops at robots.txt.....?

Thinking maybe something is wrong with my robotfile...........
this is the text in it:
User-agent: *

Disallow: /404.shtml
Disallow: 404.shtml
Disallow: espanol/404.shtml
Disallow: /espanol/404.shtml
Disallow: svenska/404.shtml
Disallow: /svenska/404.shtml
Disallow: /cgi-bin/
Disallow: /scgi-bin/

and here were inktomi explain about the robots file: http://www.inktomi.com/slurp.html
donīt understand to much, but I donīt think I done anything incorrect?

[percentages]

I would suggest deleting the blank line in robots.txt, one of the validators I used didn't appreciate the blank line, other didn't complain....but who knows?.

Next I would clean up the robots.txt so that it didn't contain superflous entries.

After that I would look at the meta tags and get rid of those that are not really necessary.

Your site should be in the Inktomi database. I can't honestly tell you why it isn't. But you sure don't need PFI.

It could be excluded for numerous reasons, as you are doing many things I would never attempt, I simply don't know the implications.

All I can say for sure is that if you clean it all up, you will be in Ink.

[helenp]

just cleaned the robots.txt
thanks

[nevetS]

I have a few web sites and my log files showed plenty of activity from slurp. No robots file in place just yet.

I searched and didn't find anything. I paid, and got awesome rankings immediately.

MSN is strangely enough starting to compete with google for traffic generated to one of my sites now - even though I have similar rankings in google.

I only submitted a couple of pages to ink. The MSN Bot has been coming frequently as well the last few days, but I don't think that is part of the paid inclusion - just a side effect.

[helenp]

Msn bot, is spidering me as well very frequently,
and I am not appearing in their search results..............

[mcavic]

Inktomi can take a long time to list your site, even after they've crawled it.

MSNBot isn't live yet in terms of results. The MSN results are currently from Slurp.

[helenp]

Hi again,

I wrote to Inktomi and they answered this:
The syntax of your /robots.txt was in error, so removing the blank line
was a good change, but that error was not blocking Slurp. Please check
our search content guidelines at
http://www.inktomi.com/products/web_search/guidelines.html.

Iīve read their guidelines, and I donīt think anything is wrong with my site,
could be because an javascript menu I have, and donīt have no script tag in, though I donīt know how it works, better study it, but I do have a sitemap with normal links on every page.

In fact I found an program that reads raw files, very good, Archive log analyzer, and I see that since 1/1 slurp spidered 7 times, but only the robots.txt file.
If iīm not indexed because of reasons said by Inktomi, the raw file would state that the spider read other files than robots.txt of my site, even if they donīt index it?

This is the stats for Inktomis spider:
SpiderReguested FileNumber of HitsBytes TransferredTm
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121213/01/2004 21:31:35
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121214/01/2004 6:49:58
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121216/01/2004 14:55:07
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 3:18:48
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 8:09:45
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 12:09:55
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 17:25:09

Only the robots.txt
...............

[percentages]

>Iīve read their guidelines, and I donīt think anything is wrong with my site

I think you got a half serious/considered answer, and half a "canned answer".

I can't see anything wrong with your site, other than the previous issues with robots.txt.

As Slurp never picked up any of your site I don't see how it can possibly be a content issue that is stopping it getting crawled/indexed. They can't reject something they have never seen!

Slurp should visit fairly regularly given the number of Ink backlinks.....I would just wait a few days and see if it is interested in taking the pages now that the robots file is clean.

Let us know....that robots.txt file shouldn't really have killed it, if it was the cause we will all have gained a little knowledge :)

[helenp]

What I was afraid of........
for several hours ago, they went in an spidered the robots.txt only
SpiderReguested FileNumber of HitsBytes TransferredTm
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt113520/01/2004 16:43:12

If they checked more pages and saw for exampel to many and inesesary metas.....wouldnīt it stat that they been on other pages?
I mean if they only spider robots.txt would it make any diferece if I made changes to the index.file for exampel, although donīt know what to change really.

If come back and spider other pages I will let you know.

And finally, if I paid them would that make them spider and index me?