Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

Inktomi Web Robot - Clarification!



5:12 am on Mar 1, 2003 (gmt 0)

10+ Year Member


I am quiet new to this forum. I have a doubt regarding Inktomi web robot.

I launched my new website few weeks back. Today I found these 2 records in my web log (inktomi web robot has visited my site).

------------------------------------------------------------ - - [28/Feb/2003:22:51:14 ] "GET /robots.txt HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/si; slurp@inktomi.com; [inktomi.com...] - - [28/Feb/2003:22:52:03 - "GET /accessibility.htm HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]


Though I did not opt for pay for inclusion option with inktomi, does this mean that inktomi web robot has located and registered my website for free indexing...?

pls clarify.


Balaji Bashyam.


5:33 am on Mar 1, 2003 (gmt 0)

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member


Welcome to WebmasterWorld [webmasterworld.com]!

Yes, quite possibly, Ink has found a link to the page shown in your log, and has indexed that page. It also tried to read your robots.txt file [robotstxt.org], but you don't have one (You might consider uploading at least a blank robots.txt to prevent the 404 errors, but you don't have to).

If you read the paid offer on the Ink site (and others), you will see that for a fee, they offer to spider and index your pages often. Though they do not emphasize it strongly, that is what the fee is for - frequency of updates, and not necessarily fee-for-inclusion.

We had another short discussion of this today in this thread [webmasterworld.com].



6:10 am on Mar 1, 2003 (gmt 0)

10+ Year Member

thanks ..Jim.


6:15 am on Mar 1, 2003 (gmt 0)

10+ Year Member

Actually what's interesting is that your page was 404 -- Not found. If those are the only 2 refrences to the Inktomi spider I don't think you're going to make the index.

Perhaps your IP was recently used for another site - and that one was in the index?


Featured Threads

Hot Threads This Week

Hot Threads This Month