Welcome to WebmasterWorld Guest from 220.127.116.11 , register , free tools , login , search , pro membership , help , library , announcements , recent posts , open posts Pubcon Platinum Sponsor 2014
Inktomi Spider Behavior what does this mean graywolf
The inktomi spider has come twice in the past 30 days and the only file it looked for was robots.txt. I don't identify it specifically in the robots file could this be why it's not looking for any other pages?
Not likely. Some bots update their robots cache regularly, regardless of their spidering schedule. andy_boyd
For a while there Slurp only came in, looked at the index and robots.txt and left. But for the last week or so Slurp has been all over one of my sites, can't get enough. Strange behaviour. helenp
I have the same problem, since 13/1 until today, inktomi slurp/si only spiders my robots.txt, and some days several times, one day 14 times. write twice to Inktomi, but nothing, no good anwers. helenp
Just noticied that gigabot also only spideres my robot.txt, My robots.txt are validated and checked and doubblechecked, the rest of spiders donīt have any problems.