Welcome to WebmasterWorld Guest from 54.225.16.10

Forum Moderators: open

Message Too Old, No Replies

Inktomisearch spider... what does it do?

     
4:43 pm on Mar 14, 2003 (gmt 0)

New User

10+ Year Member

joined:Mar 1, 2003
posts:11
votes: 0


I, like lots of people, have been hit several times lately by the inktomisearch spider (slurp). What does it do? Why aren't my other sites that are linked to my site getting spidered? Does it only check for new content, or what?
Thanks,
Theodore
10:31 pm on Mar 14, 2003 (gmt 0)

Preferred Member

10+ Year Member

joined:Dec 24, 2002
posts:464
votes: 0


Well, I never paid for inclusion, but my index page is listed.

Maybe because I am linked from a client site and he does pay for inclusion?

What does it do- well, someone with raw logs can report what files it crawls...

Maybe there is a hidden index somehere being prepared for yahoo with all this crawling.

AW

bsmithag

12:19 am on Mar 16, 2003 (gmt 0)

Inactive Member
Account Expired

 
 


I paid for Inktomi back in November. Usually it would grab my robots.txt and my index page every couple of days. Now it seems to be crawling deeper through my site.
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members