Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Crawl Rate Settings?

         

nrep

9:45 am on Apr 27, 2012 (gmt 0)

10+ Year Member



Perhaps it's something that I've not noticed before, but in WMT the crawl rate settings now say:

    Let Google optimize for my site (recommended)

    Limit Google's maximum crawl rate


I'm fairly sure that this setting didn't use to be a "limit". Am I correct in thinking that I'll get faster crawling by picking the 1st option?

I've currently got it set to a crawl rate of 10/sec, as the server can easily handle this rate.

sunnyujjawal

9:57 am on Apr 27, 2012 (gmt 0)

10+ Year Member



Googlebot employs sophisticated algorithms that determine how much to crawl each site it visits. For a vast majority of sites, it's probably best to choose the "Let Google determine my crawl rate" option, which is the default. However, if you're an advanced user or if you're facing bandwidth issues with your server, you can customize your crawl rate to the speed most optimal for your web server(s). The custom crawl rate option allows you to provide Googlebot insight to the maximum number of requests per second and the number of seconds between requests that you feel are best for your environment.