Msg#: 3801367 posted 2:40 am on Dec 6, 2008 (gmt 0)
Google has now introduced a "custom crawl rate" option in Webmaster Tools.
The custom crawl rate option allows you to provide Googlebot insight to the maximum number of requests per second and the number of seconds between requests that you feel are best for your environment...
Googlebot determines the range of crawl rate values you'll have available in Webmaster Tools. This is based on our understanding of your server's capabilities. This range may vary from one site to another and across time based on several factors.
Msg#: 3801367 posted 1:59 pm on Dec 6, 2008 (gmt 0)
i am not on blogspot and it won't allow me to change the crawl rate. The message I get is your site has been assigned special crawl rate settings. Perhaps this is due to the fact that my site is relatively new
Msg#: 3801367 posted 1:22 am on Dec 7, 2008 (gmt 0)
I can answer my own question now, it appears to take about 2 hours.
This is a great feature.
What would make it even better would be if Google would allow throttling by time of day. I would love to throttle up to the max rate on the over night hours and bring it back to normal during the day. That would be great.
Msg#: 3801367 posted 3:54 pm on Feb 17, 2009 (gmt 0)
>>>Set it to 900% and the yo-yo effect will probably go wild :)
You mean we have to avoid it or what ?
What is this crawl rates actually means as i increased it to 0.5 requests per second but no change in site traffic , but yes in google wmt i can see lot of network unreachable errors , and hosting company site is up 24 hours
So plz help me what to do
Should i change crawl rate?
What positive and negative effects 0.5 request per second have ?
and where i can get full details regarding this feauter