Massive Googlebot Crawl Rates Before Penguin

[bigcrags]

One of the sites I report on got hit by Penguin last week. I believe it was largely because of near-duplicate content they have been using, copied from one of their other sites (I had been warning them!).

In webmaster tools the site had normal crawl rates of around 500-1500 pages per day. At the start of April crawl rates increased over two weeks then jumped sharply to around 200,000 pages per day.

Has anyone else who's had a Penguin penalty noticed crawl rates like this?

[canuckseo]

I noticed huge crawl with an increase in traffic before penguin - a drop in both during the week penguin apparently rolled out, but everything so far is back to normal this week.

[Planet13]

I had a large crawl right before the March 23rd Panda update (which affected one of my sites).

It wasn't on the order of magnitude that you experienced, but it was about 6 times the normal number of pages crawled.

I think part of it was because I was having a canonical URL problem and the same page was being crawled under several different URLs.

[Sgt_Kickaxe]

I have a spike in the order of 4x normal crawl activity just before Penguin. I even wrote a post here asking if it would be wise to limit Googlebot activity because of that spike before they had announced Penguin. I was largely ignored by Penguin.

[crobb305]

I had spikes in "time spent downloading a page" but not in overall crawling.

[deadsea]

Up until about March 26th, Googlebot had been crawling 30,000 to 40,000 pages on my site per day. Then it abruptly switched to crawling a more consistent 10,000 pages per day since. You folks must have been allocated all that extra crawling that I'm not getting now.

Actually, I launched tons of international language content right about that time on various sub-domains on the same server. Its hard to tell from webmaster tools, but it may be that Googlebot is spending more time crawling that and less crawling my existing content.

[bigcrags]

Thanks for the replies. It seems to be still happening - bizarre. Googlebot is gobbling 1.5-2 Gig of bandwidth per day. Might try limiting their activity - it's not like the site's getting any traffic benefit from Google now anyway.