Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: open
I considering pay Inktomi to get include in their directory, especially knowing yahoo is going to use Inktomis database.
But what I donīt understand is why i often see inktomi spidering my site......... this spider: (Slurp/si; firstname.lastname@example.org; http://www.inktomi.com/slurp.html)
I am not included in their database, and I am not appearing in msn either, my site is comercial, so I guess it wanīt be included for free........
At the end of the day we do not know as much about Inktomi as we do about Google (...well maybe not the new Google), but Google do not list some sites, so the same must be true of Inktomi.
Inktomi have a smaller index than Google, so they must index less (hopefully that will change).
Thinking maybe something is wrong with my robotfile...........
this is the text in it:
and here were inktomi explain about the robots file: http://www.inktomi.com/slurp.html
donīt understand to much, but I donīt think I done anything incorrect?
Next I would clean up the robots.txt so that it didn't contain superflous entries.
After that I would look at the meta tags and get rid of those that are not really necessary.
Your site should be in the Inktomi database. I can't honestly tell you why it isn't. But you sure don't need PFI.
It could be excluded for numerous reasons, as you are doing many things I would never attempt, I simply don't know the implications.
All I can say for sure is that if you clean it all up, you will be in Ink.
I searched and didn't find anything. I paid, and got awesome rankings immediately.
MSN is strangely enough starting to compete with google for traffic generated to one of my sites now - even though I have similar rankings in google.
I only submitted a couple of pages to ink. The MSN Bot has been coming frequently as well the last few days, but I don't think that is part of the paid inclusion - just a side effect.
I wrote to Inktomi and they answered this:
The syntax of your /robots.txt was in error, so removing the blank line
was a good change, but that error was not blocking Slurp. Please check
our search content guidelines at
Iīve read their guidelines, and I donīt think anything is wrong with my site,
In fact I found an program that reads raw files, very good, Archive log analyzer, and I see that since 1/1 slurp spidered 7 times, but only the robots.txt file.
If iīm not indexed because of reasons said by Inktomi, the raw file would state that the spider read other files than robots.txt of my site, even if they donīt index it?
This is the stats for Inktomis spider:
SpiderReguested FileNumber of HitsBytes TransferredTm
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121213/01/2004 21:31:35
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121214/01/2004 6:49:58
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121216/01/2004 14:55:07
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 3:18:48
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 8:09:45
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 12:09:55
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 17:25:09
Only the robots.txt
I think you got a half serious/considered answer, and half a "canned answer".
I can't see anything wrong with your site, other than the previous issues with robots.txt.
As Slurp never picked up any of your site I don't see how it can possibly be a content issue that is stopping it getting crawled/indexed. They can't reject something they have never seen!
Slurp should visit fairly regularly given the number of Ink backlinks.....I would just wait a few days and see if it is interested in taking the pages now that the robots file is clean.
Let us know....that robots.txt file shouldn't really have killed it, if it was the cause we will all have gained a little knowledge :)
If they checked more pages and saw for exampel to many and inesesary metas.....wouldnīt it stat that they been on other pages?
I mean if they only spider robots.txt would it make any diferece if I made changes to the index.file for exampel, although donīt know what to change really.
If come back and spider other pages I will let you know.
And finally, if I paid them would that make them spider and index me?