Welcome to WebmasterWorld Guest from 54.161.157.73

Forum Moderators: Robert Charlton & aakk9999 & andy langton & goodroi

Message Too Old, No Replies

Massive Googlebot Crawl Rates Before Penguin

     
11:47 am on May 2, 2012 (gmt 0)

New User

10+ Year Member

joined:Mar 9, 2006
posts: 22
votes: 0


One of the sites I report on got hit by Penguin last week. I believe it was largely because of near-duplicate content they have been using, copied from one of their other sites (I had been warning them!).

In webmaster tools the site had normal crawl rates of around 500-1500 pages per day. At the start of April crawl rates increased over two weeks then jumped sharply to around 200,000 pages per day.

Has anyone else who's had a Penguin penalty noticed crawl rates like this?
3:09 pm on May 2, 2012 (gmt 0)

Junior Member

5+ Year Member

joined:Nov 6, 2008
posts: 124
votes: 0


I noticed huge crawl with an increase in traffic before penguin - a drop in both during the week penguin apparently rolled out, but everything so far is back to normal this week.
3:18 pm on May 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member planet13 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 16, 2010
posts: 3813
votes: 29


I had a large crawl right before the March 23rd Panda update (which affected one of my sites).

It wasn't on the order of magnitude that you experienced, but it was about 6 times the normal number of pages crawled.

I think part of it was because I was having a canonical URL problem and the same page was being crawled under several different URLs.
3:32 pm on May 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member

joined:Apr 14, 2010
posts:3169
votes: 0


I have a spike in the order of 4x normal crawl activity just before Penguin. I even wrote a post here asking if it would be wise to limit Googlebot activity because of that spike before they had announced Penguin. I was largely ignored by Penguin.
3:41 pm on May 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 3, 2002
posts:2579
votes: 0


I had spikes in "time spent downloading a page" but not in overall crawling.
4:25 pm on May 2, 2012 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 7, 2003
posts: 750
votes: 0


Up until about March 26th, Googlebot had been crawling 30,000 to 40,000 pages on my site per day. Then it abruptly switched to crawling a more consistent 10,000 pages per day since. You folks must have been allocated all that extra crawling that I'm not getting now.

Actually, I launched tons of international language content right about that time on various sub-domains on the same server. Its hard to tell from webmaster tools, but it may be that Googlebot is spending more time crawling that and less crawling my existing content.
8:13 am on May 4, 2012 (gmt 0)

New User

10+ Year Member

joined:Mar 9, 2006
posts: 22
votes: 0


Thanks for the replies. It seems to be still happening - bizarre. Googlebot is gobbling 1.5-2 Gig of bandwidth per day. Might try limiting their activity - it's not like the site's getting any traffic benefit from Google now anyway.