| 1:40 pm on Jun 16, 2005 (gmt 0)|
I think it's different for every site. My highest performing website that's been around for years gets spidered almost on a daily basis. While some of my other sites only get spidered occasionally. I could be wrong on this, but it seems to be true based on my own observations.
| 1:56 pm on Jun 16, 2005 (gmt 0)|
The frequency of spidering seems to be related to the Page Rank of the page in question. A PR5+ page can expect to see googlebot almost everyday. Once crawled, fresh content can be expected to appear in the results within 2-3 days.
| 2:08 pm on Jun 16, 2005 (gmt 0)|
Google pulls almost my entire site (1.6mb) daily.
| 2:10 pm on Jun 16, 2005 (gmt 0)|
Spidering of a site depends upon the longevity of its place in the SERPS and amount of new content added over a certain time period.
Basically, if the site is worth it, and it adds new content often, Googlebot moves it up the "to be refreshed" list.
| 10:25 am on Jun 19, 2005 (gmt 0)|
There is a googlebot crawling my site constantly, as far as I can tell. When I update the stats in real time the record consistently shows googlebot activity in the same minute as the update request. Last month googlebot made 102283 hits on my site and chewed through 2.69 GB of bandwidth.
| 4:51 am on Jun 24, 2005 (gmt 0)|
I have a site with a PR of 8 and I'm getting only about an average of 200 spider hits a day which equates to 6k a month. We have over 900k pages indexed in Google and add content on a daily basis. This number of spidering activities seems very low given the size of the site - any ideas? Any way to improve it?
| 5:46 am on Jun 24, 2005 (gmt 0)|
That's quite a large number of pages on your site 900k.
Is it a forum site of some sort?
| 5:47 am on Jun 24, 2005 (gmt 0)|
My site is a PR 5, but it gets crawled everyday, about 200-500 pages per day. So it depends.
| 1:33 pm on Jun 24, 2005 (gmt 0)|
It's actually a content site with all original content. So the small number seems unreasonable. I used to work for a small real estate client who would get 50k hits from spiders a month, thus my questioning.
| 2:25 pm on Jun 24, 2005 (gmt 0)|
>>How Often is a Site Spidered by Google?<<
Every Day - Never
| 2:33 pm on Jun 24, 2005 (gmt 0)|
i checked internally and apparently our webtrends data is not reliable... this number is more like 8k a day...
|Smashing Young Man|
| 2:39 pm on Jun 24, 2005 (gmt 0)|
My site just came off a penalty, has a giant goose egg for PR, and has been getting a complete crawl every day for about two weeks now.
For all that, all the pages are still residing in Google's supplemental index. Go figure.