How Often Should a Spider Come

What is the right amount of access a spider should have

[sirjeffer]

Greetings.
I have been trying to refine my Webtrends and Traffic facts web statistics and see that some spiders such as Slurp come 31,000 times a month, and others come 20. My? is

1) What is the right amount of times in order to get the best placement?

2) How do you set that value in a robots.txt file?

[mack]

The robots.txt file does not instruct spiders when to, or now often to spider your pages. The robots.txt file is only there to specify certain files or directories that you do not want spidered at all.

There isn't really any method for determining how often a sider will visit you.

Search engine bots are gettign smarter though, they try to determins how often pages update, they can them focum more of their attention towards re-indexing pages that are likely to contain fresh content.

Mack.

[sirjeffer]

So why would Yahoo/Slurp come 31,000 times in a month, and the others come 20 times? This makes no sense.

[mack]

Some bots will only crawl certain pages. (fewer clicks from homepage)

Google and yahoo tend to spider deep, taking as many pages as possible. With Google it takes a little while before it will deep index. Generaly it gets deeper on each visit.

Mack.