Welcome to WebmasterWorld Guest from 54.80.185.137

This page requires javascript to function properly. It seems that your browser does not have Javascript enabled. Please enable Javascript and press the Reload/Refresh button on your browser.

Forum Moderators: goodroi

Message Too Old, No Replies

Inktomi Spider Behavior

what does this mean

graywolf

3:02 pm on Feb 9, 2004 (gmt 0)

Senior Member

joined:Feb 28, 2002
posts:1328
votes: 0

The inktomi spider has come twice in the past 30 days and the only file it looked for was robots.txt. I don't identify it specifically in the robots file could this be why it's not looking for any other pages?

Finder

3:59 pm on Feb 16, 2004 (gmt 0)

Junior Member

joined:Aug 18, 2002
posts:131
votes: 0

Not likely. Some bots update their robots cache regularly, regardless of their spidering schedule.

andy_boyd

5:43 pm on Feb 16, 2004 (gmt 0)

Preferred Member

joined:Jan 1, 2003
posts:490
votes: 0

For a while there Slurp only came in, looked at the index and robots.txt and left. But for the last week or so Slurp has been all over one of my sites, can't get enough. Strange behaviour.

helenp

10:45 pm on Feb 16, 2004 (gmt 0)

Senior Member

joined:Nov 27, 2003
posts:1308
votes: 0

I have the same problem, since 13/1 until today,
inktomi slurp/si only spiders my robots.txt, and some days several times, one day 14 times.
write twice to Inktomi, but nothing, no good anwers.

helenp

12:31 am on Feb 17, 2004 (gmt 0)

Senior Member

joined:Nov 27, 2003
posts:1308
votes: 0

Just noticied that gigabot also only spideres my robot.txt,
My robots.txt are validated and checked and doubblechecked, the rest of spiders don�t have any problems.