Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Inktomi Web Robot - Clarification!

5:12 am on Mar 1, 2003 (gmt 0)

New User

10+ Year Member

joined:Feb 27, 2003
votes: 0


I am quiet new to this forum. I have a doubt regarding Inktomi web robot.

I launched my new website few weeks back. Today I found these 2 records in my web log (inktomi web robot has visited my site).

------------------------------------------------------------ - - [28/Feb/2003:22:51:14 ] "GET /robots.txt HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/si; slurp@inktomi.com; [inktomi.com...] - - [28/Feb/2003:22:52:03 - "GET /accessibility.htm HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]


Though I did not opt for pay for inclusion option with inktomi, does this mean that inktomi web robot has located and registered my website for free indexing...?

pls clarify.


Balaji Bashyam.

5:33 am on Mar 1, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
votes: 0


Welcome to WebmasterWorld [webmasterworld.com]!

Yes, quite possibly, Ink has found a link to the page shown in your log, and has indexed that page. It also tried to read your robots.txt file [robotstxt.org], but you don't have one (You might consider uploading at least a blank robots.txt to prevent the 404 errors, but you don't have to).

If you read the paid offer on the Ink site (and others), you will see that for a fee, they offer to spider and index your pages often. Though they do not emphasize it strongly, that is what the fee is for - frequency of updates, and not necessarily fee-for-inclusion.

We had another short discussion of this today in this thread [webmasterworld.com].


6:10 am on Mar 1, 2003 (gmt 0)

New User

10+ Year Member

joined:Feb 27, 2003
votes: 0

thanks ..Jim.
6:15 am on Mar 1, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2003
votes: 0

Actually what's interesting is that your page was 404 -- Not found. If those are the only 2 refrences to the Inktomi spider I don't think you're going to make the index.

Perhaps your IP was recently used for another site - and that one was in the index?