Google's Crawl Rate (Webmaster tools)

Some sites are crawled very slowly and have the "faster" option available

[moopy]

Hello,

I have many sites in my Google Webmaster tools account. Some of the websites are crawled very slowly, and I've noticed that some of them have the "Faster" option available, with the following message being shown:

We've detected that Googlebot is limiting the rate at which it crawls pages on your site to ensure it doesn't use too much of your server's resources. If your server can handle additional Googlebot traffic, we recommend that you choose Faster below.

After setting it, Google claims this option would return to "Normal" after 90 days.

I'd like to understand this option better. Why does Google decide that for certain sites the crawl rate would be slower?

Thanks

[SEOPTI]

Google's crawlers are adaptive: if they detect your servers slowing as page requests are received, they reduce the crawl rate.

[moopy]

I've used the past few days to check this assumption, and it is not true. I've crawled the sites myself and saw no decrease in server response times.

Any other ideas?

Thanks

[joecruise]

google crawls index page in every week but rest of pages including deep links are crawled once in the month

Regards
Joe

[tedster]

Google's crawlers are adaptive

Agreed - and in more ways than server response time. The crawl team has logic in place that allocates their resources according to how valuable crawling any particular url is. The best way to get frequent and deeper crawls is to have a website that Google sees as very important for their end users. To the degree that you do that, you will get more crawling resources.