Welcome to WebmasterWorld Guest from 54.162.76.55

Forum Moderators: open

Message Too Old, No Replies

Inktomi Web Robot - Clarification!

     
5:12 am on Mar 1, 2003 (gmt 0)

New User

10+ Year Member

joined:Feb 27, 2003
posts:17
votes: 0


Hi,

I am quiet new to this forum. I have a doubt regarding Inktomi web robot.

I launched my new website few weeks back. Today I found these 2 records in my web log (inktomi web robot has visited my site).

------------------------------------------------------------
66.196.65.26 - - [28/Feb/2003:22:51:14 ] "GET /robots.txt HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/si; slurp@inktomi.com; [inktomi.com...]

66.196.72.59 - - [28/Feb/2003:22:52:03 - "GET /accessibility.htm HTTP/1.0" 404 5908 "-" "Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; [inktomi.com...]

------------------------------------------------------------

Though I did not opt for pay for inclusion option with inktomi, does this mean that inktomi web robot has located and registered my website for free indexing...?

pls clarify.

Thanks.

Balaji Bashyam.

5:33 am on Mar 1, 2003 (gmt 0)

Senior Member

WebmasterWorld Senior Member jdmorgan is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 31, 2002
posts:25430
votes: 0


Balaji,

Welcome to WebmasterWorld [webmasterworld.com]!

Yes, quite possibly, Ink has found a link to the page shown in your log, and has indexed that page. It also tried to read your robots.txt file [robotstxt.org], but you don't have one (You might consider uploading at least a blank robots.txt to prevent the 404 errors, but you don't have to).

If you read the paid offer on the Ink site (and others), you will see that for a fee, they offer to spider and index your pages often. Though they do not emphasize it strongly, that is what the fee is for - frequency of updates, and not necessarily fee-for-inclusion.

We had another short discussion of this today in this thread [webmasterworld.com].

Jim

6:10 am on Mar 1, 2003 (gmt 0)

New User

10+ Year Member

joined:Feb 27, 2003
posts:17
votes: 0


thanks ..Jim.
6:15 am on Mar 1, 2003 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 20, 2003
posts:151
votes: 0


Actually what's interesting is that your page was 404 -- Not found. If those are the only 2 refrences to the Inktomi spider I don't think you're going to make the index.

Perhaps your IP was recently used for another site - and that one was in the index?