Forum Moderators: goodroi

Message Too Old, No Replies

How Often Should a Spider Come

What is the right amount of access a spider should have

         

sirjeffer

6:50 pm on Aug 22, 2005 (gmt 0)

10+ Year Member



Greetings.
I have been trying to refine my Webtrends and Traffic facts web statistics and see that some spiders such as Slurp come 31,000 times a month, and others come 20. My? is

1) What is the right amount of times in order to get the best placement?

2) How do you set that value in a robots.txt file?

mack

6:55 pm on Aug 22, 2005 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



The robots.txt file does not instruct spiders when to, or now often to spider your pages. The robots.txt file is only there to specify certain files or directories that you do not want spidered at all.

There isn't really any method for determining how often a sider will visit you.

Search engine bots are gettign smarter though, they try to determins how often pages update, they can them focum more of their attention towards re-indexing pages that are likely to contain fresh content.

Mack.

sirjeffer

8:00 pm on Aug 22, 2005 (gmt 0)

10+ Year Member



So why would Yahoo/Slurp come 31,000 times in a month, and the others come 20 times? This makes no sense.

mack

11:00 pm on Aug 22, 2005 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



Some bots will only crawl certain pages. (fewer clicks from homepage)

Google and yahoo tend to spider deep, taking as many pages as possible. With Google it takes a little while before it will deep index. Generaly it gets deeper on each visit.

Mack.