homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

How Often is a Site Spidered by Google?

 11:06 pm on Jun 15, 2005 (gmt 0)

I'm just wondering, is there a set frequency with which the google spiders or any SE spiders for that matter will come and pay a visit to your site?

Also, if I was to add new pages to my site and the site was spidered, how soon thereafter can I expect to see them in the SE?

Thank you



 1:40 pm on Jun 16, 2005 (gmt 0)

I think it's different for every site. My highest performing website that's been around for years gets spidered almost on a daily basis. While some of my other sites only get spidered occasionally. I could be wrong on this, but it seems to be true based on my own observations.


 1:56 pm on Jun 16, 2005 (gmt 0)

The frequency of spidering seems to be related to the Page Rank of the page in question. A PR5+ page can expect to see googlebot almost everyday. Once crawled, fresh content can be expected to appear in the results within 2-3 days.


 2:08 pm on Jun 16, 2005 (gmt 0)

Google pulls almost my entire site (1.6mb) daily.


 2:10 pm on Jun 16, 2005 (gmt 0)

Spidering of a site depends upon the longevity of its place in the SERPS and amount of new content added over a certain time period.

Basically, if the site is worth it, and it adds new content often, Googlebot moves it up the "to be refreshed" list.

Billy Batson

 10:25 am on Jun 19, 2005 (gmt 0)

There is a googlebot crawling my site constantly, as far as I can tell. When I update the stats in real time the record consistently shows googlebot activity in the same minute as the update request. Last month googlebot made 102283 hits on my site and chewed through 2.69 GB of bandwidth.


 4:51 am on Jun 24, 2005 (gmt 0)

I have a site with a PR of 8 and I'm getting only about an average of 200 spider hits a day which equates to 6k a month. We have over 900k pages indexed in Google and add content on a daily basis. This number of spidering activities seems very low given the size of the site - any ideas? Any way to improve it?



 5:46 am on Jun 24, 2005 (gmt 0)


That's quite a large number of pages on your site 900k.

Is it a forum site of some sort?


 5:47 am on Jun 24, 2005 (gmt 0)

My site is a PR 5, but it gets crawled everyday, about 200-500 pages per day. So it depends.


 1:33 pm on Jun 24, 2005 (gmt 0)

It's actually a content site with all original content. So the small number seems unreasonable. I used to work for a small real estate client who would get 50k hits from spiders a month, thus my questioning.


 2:25 pm on Jun 24, 2005 (gmt 0)

>>How Often is a Site Spidered by Google?<<

Every Day - Never


 2:33 pm on Jun 24, 2005 (gmt 0)

i checked internally and apparently our webtrends data is not reliable... this number is more like 8k a day...

Smashing Young Man

 2:39 pm on Jun 24, 2005 (gmt 0)

My site just came off a penalty, has a giant goose egg for PR, and has been getting a complete crawl every day for about two weeks now.

For all that, all the pages are still residing in Google's supplemental index. Go figure.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved