Msg#: 4448398 posted 11:47 am on May 2, 2012 (gmt 0)
One of the sites I report on got hit by Penguin last week. I believe it was largely because of near-duplicate content they have been using, copied from one of their other sites (I had been warning them!).
In webmaster tools the site had normal crawl rates of around 500-1500 pages per day. At the start of April crawl rates increased over two weeks then jumped sharply to around 200,000 pages per day.
Has anyone else who's had a Penguin penalty noticed crawl rates like this?
Msg#: 4448398 posted 3:32 pm on May 2, 2012 (gmt 0)
I have a spike in the order of 4x normal crawl activity just before Penguin. I even wrote a post here asking if it would be wise to limit Googlebot activity because of that spike before they had announced Penguin. I was largely ignored by Penguin.
Msg#: 4448398 posted 4:25 pm on May 2, 2012 (gmt 0)
Up until about March 26th, Googlebot had been crawling 30,000 to 40,000 pages on my site per day. Then it abruptly switched to crawling a more consistent 10,000 pages per day since. You folks must have been allocated all that extra crawling that I'm not getting now.
Actually, I launched tons of international language content right about that time on various sub-domains on the same server. Its hard to tell from webmaster tools, but it may be that Googlebot is spending more time crawling that and less crawling my existing content.
Msg#: 4448398 posted 8:13 am on May 4, 2012 (gmt 0)
Thanks for the replies. It seems to be still happening - bizarre. Googlebot is gobbling 1.5-2 Gig of bandwidth per day. Might try limiting their activity - it's not like the site's getting any traffic benefit from Google now anyway.