rowan194 - 2:01 am on Jun 18, 2011 (gmt 0) [edited by: rowan194 at 2:28 am (utc) on Jun 18, 2011]
It's GWT where I'm setting the custom crawl rate. Google specifically ignores the crawl-delay robots.txt directive - GWT even tells you they've ignored that line. :)
I think I may have found the problem - GWT seems to treat domain.com and www.domain.com as separate entities. Only one is listed as confirmed in my case, and googlebot is predominantly fetching from the *other*.
I don't recall it used to be like this, or if they made that change why they didn't offer some way to tie the two entities together (the verification filename is exactly the same for both...)
I've changed the crawl rate on the other one.
So now I have to mark my calendar to update two "domains" every 90 days! >:(
edit: now that domain.com has been verified it does seem to have tied it to www.domain.com. I've set the preferred domain to domain.com as that's the one that has the most indexed pages in G. Would be nice if they (a) forwarded system generated notifications to your email address (rather than requiring you to log into GWT to see them) and (b) proactively invite you to review your settings when some new feature is added that may affect your sites.
[edited by: rowan194 at 2:28 am (utc) on Jun 18, 2011]