homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
Forum Library, Charter, Moderator: open

Paid Inclusion Engines and Topics Forum

  posting off  
inktomi spiders me, why?

 11:05 pm on Jan 17, 2004 (gmt 0)

I considering pay Inktomi to get include in their directory, especially knowing yahoo is going to use Inktomis database.

But what I donīt understand is why i often see inktomi spidering my site......... this spider: (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

I am not included in their database, and I am not appearing in msn either, my site is comercial, so I guess it wanīt be included for free........



 11:22 pm on Jan 17, 2004 (gmt 0)

Inktomi is not a pay for inclusion search engine. It is a crawler based search engine (like Google). If you have good links to your site, you will be in Inktomi.

ALl the payment to Inktomi does is guarantee that you get crawled every 48hrs


 11:26 pm on Jan 17, 2004 (gmt 0)

Yes, but I canīt find my domain on their database, nor in msn or hotbot........... I had good links for a year.
If so, why am I not included, does it take very, very long?
If so, that does mean as well that there are no need to pay Inktomi to make sure you will be in Yahoos websearch when they switch to Inktomi?



 11:31 pm on Jan 17, 2004 (gmt 0)

Can you sticky mail me the URL?


 11:42 pm on Jan 17, 2004 (gmt 0)

just sent the sticky,
hope I did it correctly,
ever done before


 11:55 pm on Jan 17, 2004 (gmt 0)

You are not in the Inktomi index as far as I can see. There are a lot of sites that link to you that are in the Ink index --> maybe wait and see what happens. Not sure of reasons, but maybe others with more experience on this can comment.

At the end of the day we do not know as much about Inktomi as we do about Google (...well maybe not the new Google), but Google do not list some sites, so the same must be true of Inktomi.

Inktomi have a smaller index than Google, so they must index less (hopefully that will change).


 12:21 am on Jan 18, 2004 (gmt 0)
Trying to read the rawlog, because everytime I saw inktomi in my stats, (not to good) it always stated my robots.txt file.
As far as I can see, reading it, itīs dificult without a program, it stops at robots.txt.....?

Thinking maybe something is wrong with my robotfile...........
this is the text in it:
User-agent: *

Disallow: /404.shtml
Disallow: 404.shtml
Disallow: espanol/404.shtml
Disallow: /espanol/404.shtml
Disallow: svenska/404.shtml
Disallow: /svenska/404.shtml
Disallow: /cgi-bin/
Disallow: /scgi-bin/

and here were inktomi explain about the robots file: http://www.inktomi.com/slurp.html
donīt understand to much, but I donīt think I done anything incorrect?


 8:03 am on Jan 18, 2004 (gmt 0)

I would suggest deleting the blank line in robots.txt, one of the validators I used didn't appreciate the blank line, other didn't complain....but who knows?.

Next I would clean up the robots.txt so that it didn't contain superflous entries.

After that I would look at the meta tags and get rid of those that are not really necessary.

Your site should be in the Inktomi database. I can't honestly tell you why it isn't. But you sure don't need PFI.

It could be excluded for numerous reasons, as you are doing many things I would never attempt, I simply don't know the implications.

All I can say for sure is that if you clean it all up, you will be in Ink.


 10:49 am on Jan 18, 2004 (gmt 0)

just cleaned the robots.txt


 7:19 pm on Jan 18, 2004 (gmt 0)

I have a few web sites and my log files showed plenty of activity from slurp. No robots file in place just yet.

I searched and didn't find anything. I paid, and got awesome rankings immediately.

MSN is strangely enough starting to compete with google for traffic generated to one of my sites now - even though I have similar rankings in google.

I only submitted a couple of pages to ink. The MSN Bot has been coming frequently as well the last few days, but I don't think that is part of the paid inclusion - just a side effect.


 12:50 am on Jan 19, 2004 (gmt 0)

Msn bot, is spidering me as well very frequently,
and I am not appearing in their search results..............


 6:19 am on Jan 19, 2004 (gmt 0)

Inktomi can take a long time to list your site, even after they've crawled it.

MSNBot isn't live yet in terms of results. The MSN results are currently from Slurp.


 10:47 pm on Jan 19, 2004 (gmt 0)
Hi again,

I wrote to Inktomi and they answered this:
The syntax of your /robots.txt was in error, so removing the blank line
was a good change, but that error was not blocking Slurp. Please check
our search content guidelines at

Iīve read their guidelines, and I donīt think anything is wrong with my site,
could be because an javascript menu I have, and donīt have no script tag in, though I donīt know how it works, better study it, but I do have a sitemap with normal links on every page.

In fact I found an program that reads raw files, very good, Archive log analyzer, and I see that since 1/1 slurp spidered 7 times, but only the robots.txt file.
If iīm not indexed because of reasons said by Inktomi, the raw file would state that the spider read other files than robots.txt of my site, even if they donīt index it?

This is the stats for Inktomis spider:
SpiderReguested FileNumber of HitsBytes TransferredTm
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121213/01/2004 21:31:35
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121214/01/2004 6:49:58
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121216/01/2004 14:55:07
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 3:18:48
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 8:09:45
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 12:09:55
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt121217/01/2004 17:25:09

Only the robots.txt


 7:57 am on Jan 20, 2004 (gmt 0)

>Iīve read their guidelines, and I donīt think anything is wrong with my site

I think you got a half serious/considered answer, and half a "canned answer".

I can't see anything wrong with your site, other than the previous issues with robots.txt.

As Slurp never picked up any of your site I don't see how it can possibly be a content issue that is stopping it getting crawled/indexed. They can't reject something they have never seen!

Slurp should visit fairly regularly given the number of Ink backlinks.....I would just wait a few days and see if it is interested in taking the pages now that the robots file is clean.

Let us know....that robots.txt file shouldn't really have killed it, if it was the cause we will all have gained a little knowledge :)


 11:50 pm on Jan 20, 2004 (gmt 0)

What I was afraid of........
for several hours ago, they went in an spidered the robots.txt only
SpiderReguested FileNumber of HitsBytes TransferredTm
Inktomi (Hotbot & AOL & GoTo & MSN & Iwon)/robots.txt113520/01/2004 16:43:12

If they checked more pages and saw for exampel to many and inesesary metas.....wouldnīt it stat that they been on other pages?
I mean if they only spider robots.txt would it make any diferece if I made changes to the index.file for exampel, although donīt know what to change really.

If come back and spider other pages I will let you know.

And finally, if I paid them would that make them spider and index me?

Global Options:
 top home search open messages active posts  

Home / Forums Index / Advertising / Paid Inclusion Engines and Topics
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved