Inktomi Web Robot - Clarification!

[bashyam]

Hi,

I am quiet new to this forum. I have a doubt regarding Inktomi web robot.

I launched my new website few weeks back. Today I found these 2 records in my web log (inktomi web robot has visited my site).

------------------------------------------------------------
66.196.65.26 - - [28/Feb/2003:22:51:14 ] "GET /robots.txt HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/si; [e-mail]; [inktomi.com...]

66.196.72.59 - - [28/Feb/2003:22:52:03 - "GET /accessibility.htm HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/cat; [e-mail]; [inktomi.com...]

------------------------------------------------------------

Though I did not opt for pay for inclusion option with inktomi, does this mean that inktomi web robot has located and registered my website for free indexing...?

pls clarify.

Thanks.

Balaji Bashyam.

[jdMorgan]

Balaji,

Welcome to WebmasterWorld [webmasterworld.com]!

Yes, quite possibly, Ink has found a link to the page shown in your log, and has indexed that page. It also tried to read your robots.txt file [robotstxt.org], but you don't have one (You might consider uploading at least a blank robots.txt to prevent the 404 errors, but you don't have to).

If you read the paid offer on the Ink site (and others), you will see that for a fee, they offer to spider and index your pages often. Though they do not emphasize it strongly, that is what the fee is for - frequency of updates, and not necessarily fee-for-inclusion.

We had another short discussion of this today in this thread [webmasterworld.com].

Jim

[bashyam]

thanks ..Jim.

[daroz]

Actually what's interesting is that your page was 404 -- Not found. If those are the only 2 refrences to the Inktomi spider I don't think you're going to make the index.

Perhaps your IP was recently used for another site - and that one was in the index?