Welcome to WebmasterWorld Guest from 54.163.168.15

Massive Googlebot Crawl Rates Before Penguin

   
11:47 am on May 2, 2012 (gmt 0)

5+ Year Member



One of the sites I report on got hit by Penguin last week. I believe it was largely because of near-duplicate content they have been using, copied from one of their other sites (I had been warning them!).

In webmaster tools the site had normal crawl rates of around 500-1500 pages per day. At the start of April crawl rates increased over two weeks then jumped sharply to around 200,000 pages per day.

Has anyone else who's had a Penguin penalty noticed crawl rates like this?
3:09 pm on May 2, 2012 (gmt 0)

5+ Year Member



I noticed huge crawl with an increase in traffic before penguin - a drop in both during the week penguin apparently rolled out, but everything so far is back to normal this week.
3:18 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member planet13 is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



I had a large crawl right before the March 23rd Panda update (which affected one of my sites).

It wasn't on the order of magnitude that you experienced, but it was about 6 times the normal number of pages crawled.

I think part of it was because I was having a canonical URL problem and the same page was being crawled under several different URLs.
3:32 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member



I have a spike in the order of 4x normal crawl activity just before Penguin. I even wrote a post here asking if it would be wise to limit Googlebot activity because of that spike before they had announced Penguin. I was largely ignored by Penguin.
3:41 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member crobb305 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I had spikes in "time spent downloading a page" but not in overall crawling.
4:25 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Up until about March 26th, Googlebot had been crawling 30,000 to 40,000 pages on my site per day. Then it abruptly switched to crawling a more consistent 10,000 pages per day since. You folks must have been allocated all that extra crawling that I'm not getting now.

Actually, I launched tons of international language content right about that time on various sub-domains on the same server. Its hard to tell from webmaster tools, but it may be that Googlebot is spending more time crawling that and less crawling my existing content.
8:13 am on May 4, 2012 (gmt 0)

5+ Year Member



Thanks for the replies. It seems to be still happening - bizarre. Googlebot is gobbling 1.5-2 Gig of bandwidth per day. Might try limiting their activity - it's not like the site's getting any traffic benefit from Google now anyway.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month