Welcome to WebmasterWorld Guest from 107.20.54.98

Message Too Old, No Replies

How Often is a Site Spidered by Google?

     
11:06 pm on Jun 15, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 13, 2003
posts:69
votes: 0


I'm just wondering, is there a set frequency with which the google spiders or any SE spiders for that matter will come and pay a visit to your site?

Also, if I was to add new pages to my site and the site was spidered, how soon thereafter can I expect to see them in the SE?

Thank you

1:40 pm on June 16, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:May 5, 2005
posts:111
votes: 0


I think it's different for every site. My highest performing website that's been around for years gets spidered almost on a daily basis. While some of my other sites only get spidered occasionally. I could be wrong on this, but it seems to be true based on my own observations.
1:56 pm on June 16, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 4, 2002
posts:1687
votes: 0


The frequency of spidering seems to be related to the Page Rank of the page in question. A PR5+ page can expect to see googlebot almost everyday. Once crawled, fresh content can be expected to appear in the results within 2-3 days.
2:08 pm on June 16, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 4, 2005
posts:179
votes: 0


Google pulls almost my entire site (1.6mb) daily.
2:10 pm on June 16, 2005 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 22, 2002
posts:1001
votes: 0


Spidering of a site depends upon the longevity of its place in the SERPS and amount of new content added over a certain time period.

Basically, if the site is worth it, and it adds new content often, Googlebot moves it up the "to be refreshed" list.

10:25 am on June 19, 2005 (gmt 0)

New User

10+ Year Member

joined:May 30, 2005
posts:21
votes: 0


There is a googlebot crawling my site constantly, as far as I can tell. When I update the stats in real time the record consistently shows googlebot activity in the same minute as the update request. Last month googlebot made 102283 hits on my site and chewed through 2.69 GB of bandwidth.
4:51 am on June 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 24, 2005
posts:59
votes: 0


I have a site with a PR of 8 and I'm getting only about an average of 200 spider hits a day which equates to 6k a month. We have over 900k pages indexed in Google and add content on a daily basis. This number of spidering activities seems very low given the size of the site - any ideas? Any way to improve it?

Vic

5:46 am on June 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Aug 13, 2003
posts:69
votes: 0


vicyankees

That's quite a large number of pages on your site 900k.

Is it a forum site of some sort?

5:47 am on June 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 27, 2002
posts:55
votes: 0


My site is a PR 5, but it gets crawled everyday, about 200-500 pages per day. So it depends.
1:33 pm on June 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 24, 2005
posts:59
votes: 0


It's actually a content site with all original content. So the small number seems unreasonable. I used to work for a small real estate client who would get 50k hits from spiders a month, thus my questioning.
2:25 pm on June 24, 2005 (gmt 0)

Junior Member

joined:Nov 1, 2003
posts:101
votes: 0


>>How Often is a Site Spidered by Google?<<

Every Day - Never

2:33 pm on June 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:June 24, 2005
posts:59
votes: 0


i checked internally and apparently our webtrends data is not reliable... this number is more like 8k a day...
2:39 pm on June 24, 2005 (gmt 0)

Junior Member

10+ Year Member

joined:Jan 19, 2005
posts:73
votes: 0


My site just came off a penalty, has a giant goose egg for PR, and has been getting a complete crawl every day for about two weeks now.

For all that, all the pages are still residing in Google's supplemental index. Go figure.