rowan194 - 1:17 am on Jun 14, 2011 (gmt 0)
[Apologies if this is not the correct forum - it's about a bot behaving badly, but it may be more appropriate in a Google specific area. I couldn't find an appropriate one.]
I set my custom crawl rate to 0.1/sec (10 second delay between fetches) a couple of months ago, but Googlebot is still fetching at a much faster rate.
I pulled some random samples from recent minutes:
21:02 - 27 fetches
23:29 - 33 fetches
06:48 - 45 fetches
10:52 - 47 fetches
11:02 - 39 fetches
And to be sure it wasn't a longer term thing, I counted the fetches over the last complete day (a 24 hour period) - 57792 fetches, or an average of 40.1 per minute.
10 seconds between fetches should be a maximum of 6 per minute... so why isn't my setting being respected? I just changed it to 15 sec between fetches to see if that gives something a kick.
I've set the custom crawl rate because my site has millions of pages and Google's search referrals are not proportional to the rate at which it's indexing those. The infamous cap? (For several months I was consistently receiving around 30,000 referrals per day, even though Google's count of indexed pages had grown 20-30 times over that period.)
Side note, does anyone else think it's extremely arrogant for Google to reset the requested custom rate after 90 days? Remind me then to review my decision by all means, but don't just quietly drop it and go back to the default, forcing me to log in and change the setting to what I actually want... >:(