Welcome to WebmasterWorld Guest from 54.234.8.146

Message Too Old, No Replies

Massive Googlebot Crawl Rates Before Penguin

     

bigcrags

11:47 am on May 2, 2012 (gmt 0)

5+ Year Member



One of the sites I report on got hit by Penguin last week. I believe it was largely because of near-duplicate content they have been using, copied from one of their other sites (I had been warning them!).

In webmaster tools the site had normal crawl rates of around 500-1500 pages per day. At the start of April crawl rates increased over two weeks then jumped sharply to around 200,000 pages per day.

Has anyone else who's had a Penguin penalty noticed crawl rates like this?

canuckseo

3:09 pm on May 2, 2012 (gmt 0)

5+ Year Member



I noticed huge crawl with an increase in traffic before penguin - a drop in both during the week penguin apparently rolled out, but everything so far is back to normal this week.

Planet13

3:18 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member planet13 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



I had a large crawl right before the March 23rd Panda update (which affected one of my sites).

It wasn't on the order of magnitude that you experienced, but it was about 6 times the normal number of pages crawled.

I think part of it was because I was having a canonical URL problem and the same page was being crawled under several different URLs.

Sgt_Kickaxe

3:32 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member



I have a spike in the order of 4x normal crawl activity just before Penguin. I even wrote a post here asking if it would be wise to limit Googlebot activity because of that spike before they had announced Penguin. I was largely ignored by Penguin.

crobb305

3:41 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member crobb305 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



I had spikes in "time spent downloading a page" but not in overall crawling.

deadsea

4:25 pm on May 2, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Up until about March 26th, Googlebot had been crawling 30,000 to 40,000 pages on my site per day. Then it abruptly switched to crawling a more consistent 10,000 pages per day since. You folks must have been allocated all that extra crawling that I'm not getting now.

Actually, I launched tons of international language content right about that time on various sub-domains on the same server. Its hard to tell from webmaster tools, but it may be that Googlebot is spending more time crawling that and less crawling my existing content.

bigcrags

8:13 am on May 4, 2012 (gmt 0)

5+ Year Member



Thanks for the replies. It seems to be still happening - bizarre. Googlebot is gobbling 1.5-2 Gig of bandwidth per day. Might try limiting their activity - it's not like the site's getting any traffic benefit from Google now anyway.
 

Featured Threads

Hot Threads This Week

Hot Threads This Month