Welcome to WebmasterWorld Guest from 54.147.10.72

Message Too Old, No Replies

How Often is a Site Spidered by Google?

     

notbob

11:06 pm on Jun 15, 2005 (gmt 0)

10+ Year Member



I'm just wondering, is there a set frequency with which the google spiders or any SE spiders for that matter will come and pay a visit to your site?

Also, if I was to add new pages to my site and the site was spidered, how soon thereafter can I expect to see them in the SE?

Thank you

nanotopia

1:40 pm on Jun 16, 2005 (gmt 0)

10+ Year Member



I think it's different for every site. My highest performing website that's been around for years gets spidered almost on a daily basis. While some of my other sites only get spidered occasionally. I could be wrong on this, but it seems to be true based on my own observations.

Stefan

1:56 pm on Jun 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



The frequency of spidering seems to be related to the Page Rank of the page in question. A PR5+ page can expect to see googlebot almost everyday. Once crawled, fresh content can be expected to appear in the results within 2-3 days.

thecityofgold2005

2:08 pm on Jun 16, 2005 (gmt 0)

10+ Year Member



Google pulls almost my entire site (1.6mb) daily.

Sanenet

2:10 pm on Jun 16, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Spidering of a site depends upon the longevity of its place in the SERPS and amount of new content added over a certain time period.

Basically, if the site is worth it, and it adds new content often, Googlebot moves it up the "to be refreshed" list.

Billy Batson

10:25 am on Jun 19, 2005 (gmt 0)

10+ Year Member



There is a googlebot crawling my site constantly, as far as I can tell. When I update the stats in real time the record consistently shows googlebot activity in the same minute as the update request. Last month googlebot made 102283 hits on my site and chewed through 2.69 GB of bandwidth.

vicyankees

4:51 am on Jun 24, 2005 (gmt 0)

10+ Year Member



I have a site with a PR of 8 and I'm getting only about an average of 200 spider hits a day which equates to 6k a month. We have over 900k pages indexed in Google and add content on a daily basis. This number of spidering activities seems very low given the size of the site - any ideas? Any way to improve it?

Vic

notbob

5:46 am on Jun 24, 2005 (gmt 0)

10+ Year Member



vicyankees

That's quite a large number of pages on your site 900k.

Is it a forum site of some sort?

estaquieto

5:47 am on Jun 24, 2005 (gmt 0)

10+ Year Member



My site is a PR 5, but it gets crawled everyday, about 200-500 pages per day. So it depends.

vicyankees

1:33 pm on Jun 24, 2005 (gmt 0)

10+ Year Member



It's actually a content site with all original content. So the small number seems unreasonable. I used to work for a small real estate client who would get 50k hits from spiders a month, thus my questioning.

Panacea

2:25 pm on Jun 24, 2005 (gmt 0)



>>How Often is a Site Spidered by Google?<<

Every Day - Never

vicyankees

2:33 pm on Jun 24, 2005 (gmt 0)

10+ Year Member



i checked internally and apparently our webtrends data is not reliable... this number is more like 8k a day...

Smashing Young Man

2:39 pm on Jun 24, 2005 (gmt 0)

10+ Year Member



My site just came off a penalty, has a giant goose egg for PR, and has been getting a complete crawl every day for about two weeks now.

For all that, all the pages are still residing in Google's supplemental index. Go figure.

 

Featured Threads

Hot Threads This Week

Hot Threads This Month