| 3:09 pm on May 2, 2012 (gmt 0)|
I noticed huge crawl with an increase in traffic before penguin - a drop in both during the week penguin apparently rolled out, but everything so far is back to normal this week.
| 3:18 pm on May 2, 2012 (gmt 0)|
I had a large crawl right before the March 23rd Panda update (which affected one of my sites).
It wasn't on the order of magnitude that you experienced, but it was about 6 times the normal number of pages crawled.
I think part of it was because I was having a canonical URL problem and the same page was being crawled under several different URLs.
| 3:32 pm on May 2, 2012 (gmt 0)|
I have a spike in the order of 4x normal crawl activity just before Penguin. I even wrote a post here asking if it would be wise to limit Googlebot activity because of that spike before they had announced Penguin. I was largely ignored by Penguin.
| 3:41 pm on May 2, 2012 (gmt 0)|
I had spikes in "time spent downloading a page" but not in overall crawling.
| 4:25 pm on May 2, 2012 (gmt 0)|
Up until about March 26th, Googlebot had been crawling 30,000 to 40,000 pages on my site per day. Then it abruptly switched to crawling a more consistent 10,000 pages per day since. You folks must have been allocated all that extra crawling that I'm not getting now.
Actually, I launched tons of international language content right about that time on various sub-domains on the same server. Its hard to tell from webmaster tools, but it may be that Googlebot is spending more time crawling that and less crawling my existing content.
| 8:13 am on May 4, 2012 (gmt 0)|
Thanks for the replies. It seems to be still happening - bizarre. Googlebot is gobbling 1.5-2 Gig of bandwidth per day. Might try limiting their activity - it's not like the site's getting any traffic benefit from Google now anyway.