Page is a not externally linkable
- Google
-- Google SEO News and Discussion
---- How long for Google to spider 29.5 million pages?


irldonalb - 10:06 am on Sep 11, 2007 (gmt 0)


Am I right in saying Google will allocate a certain amount of crawl time for each website depending on PageRank, back links, etc. Googlebot will spider your site for the allocated time. New sites occasionally get a temporary surge in rankings and crawl rate. You might be experiencing this.

If (and it’s a big if) your going to achieve this, you want Google to visit different sections of your website on each crawl. If you’re using Google sitemaps I’d advise you to set the change frequency to never so Google won’t re-crawl the same page. In addition I’d have a random hot section / category on the homepage each day. This might send the spider in a different direction on each crawl. It’s worked for me in the pass but on nothing of this magnitude.


Thread source:: http://www.webmasterworld.com/google/3446746.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com