How Often is a Site Spidered by Google?

[notbob]

I'm just wondering, is there a set frequency with which the google spiders or any SE spiders for that matter will come and pay a visit to your site?

Also, if I was to add new pages to my site and the site was spidered, how soon thereafter can I expect to see them in the SE?

Thank you

[nanotopia]

I think it's different for every site. My highest performing website that's been around for years gets spidered almost on a daily basis. While some of my other sites only get spidered occasionally. I could be wrong on this, but it seems to be true based on my own observations.

[Stefan]

The frequency of spidering seems to be related to the Page Rank of the page in question. A PR5+ page can expect to see googlebot almost everyday. Once crawled, fresh content can be expected to appear in the results within 2-3 days.

[thecityofgold2005]

Google pulls almost my entire site (1.6mb) daily.

[Sanenet]

Spidering of a site depends upon the longevity of its place in the SERPS and amount of new content added over a certain time period.

Basically, if the site is worth it, and it adds new content often, Googlebot moves it up the "to be refreshed" list.

[Billy Batson]

There is a googlebot crawling my site constantly, as far as I can tell. When I update the stats in real time the record consistently shows googlebot activity in the same minute as the update request. Last month googlebot made 102283 hits on my site and chewed through 2.69 GB of bandwidth.

[vicyankees]

I have a site with a PR of 8 and I'm getting only about an average of 200 spider hits a day which equates to 6k a month. We have over 900k pages indexed in Google and add content on a daily basis. This number of spidering activities seems very low given the size of the site - any ideas? Any way to improve it?

Vic

[notbob]

vicyankees

That's quite a large number of pages on your site 900k.

Is it a forum site of some sort?

[estaquieto]

My site is a PR 5, but it gets crawled everyday, about 200-500 pages per day. So it depends.

[vicyankees]

It's actually a content site with all original content. So the small number seems unreasonable. I used to work for a small real estate client who would get 50k hits from spiders a month, thus my questioning.

[Panacea]

>>How Often is a Site Spidered by Google?<<

Every Day - Never

[vicyankees]

i checked internally and apparently our webtrends data is not reliable... this number is more like 8k a day...

[Smashing Young Man]

My site just came off a penalty, has a giant goose egg for PR, and has been getting a complete crawl every day for about two weeks now.

For all that, all the pages are still residing in Google's supplemental index. Go figure.