Inktomi: weird Slurp behavior

fetches robots.txt repeatedly, but nothing else

[Finder]

For the last week, Slurp has been taking a look at robots.txt every day -- but not anything else on the site. Then today it got the file, waited two minutes and got it again, waited an hour and got it again, then another hour and got it a fourth time.

My site is a free listing. Any idea why it would need to check the robots file so often? And without indexing the site? I used the validator and my file is fine. Only my images dir is protected.

[jdMorgan]

Finder,

Slurp is weird...

It's been trying to fetch a couple of pages that have been 404 on my site for over a year. One used to have incoming links, but the other never has had any - so they must be "remembering" it from past spidering sessions. I've also seen exactly the behaviour you describe above. It may happen when they find a link to your site, and just want to check to see if they should spider that link later. Seems like a very wasteful way to do it, though.

Jim