homepage Welcome to WebmasterWorld Guest from 54.211.97.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Massive Googlebot Crawl Rates Before Penguin
bigcrags

5+ Year Member



 
Msg#: 4448398 posted 11:47 am on May 2, 2012 (gmt 0)

One of the sites I report on got hit by Penguin last week. I believe it was largely because of near-duplicate content they have been using, copied from one of their other sites (I had been warning them!).

In webmaster tools the site had normal crawl rates of around 500-1500 pages per day. At the start of April crawl rates increased over two weeks then jumped sharply to around 200,000 pages per day.

Has anyone else who's had a Penguin penalty noticed crawl rates like this?

 

canuckseo

5+ Year Member



 
Msg#: 4448398 posted 3:09 pm on May 2, 2012 (gmt 0)

I noticed huge crawl with an increase in traffic before penguin - a drop in both during the week penguin apparently rolled out, but everything so far is back to normal this week.

Planet13

WebmasterWorld Senior Member planet13 us a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month



 
Msg#: 4448398 posted 3:18 pm on May 2, 2012 (gmt 0)

I had a large crawl right before the March 23rd Panda update (which affected one of my sites).

It wasn't on the order of magnitude that you experienced, but it was about 6 times the normal number of pages crawled.

I think part of it was because I was having a canonical URL problem and the same page was being crawled under several different URLs.

Sgt_Kickaxe

WebmasterWorld Senior Member sgt_kickaxe us a WebmasterWorld Top Contributor of All Time



 
Msg#: 4448398 posted 3:32 pm on May 2, 2012 (gmt 0)

I have a spike in the order of 4x normal crawl activity just before Penguin. I even wrote a post here asking if it would be wise to limit Googlebot activity because of that spike before they had announced Penguin. I was largely ignored by Penguin.

crobb305

WebmasterWorld Senior Member crobb305 us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4448398 posted 3:41 pm on May 2, 2012 (gmt 0)

I had spikes in "time spent downloading a page" but not in overall crawling.

deadsea

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 4448398 posted 4:25 pm on May 2, 2012 (gmt 0)

Up until about March 26th, Googlebot had been crawling 30,000 to 40,000 pages on my site per day. Then it abruptly switched to crawling a more consistent 10,000 pages per day since. You folks must have been allocated all that extra crawling that I'm not getting now.

Actually, I launched tons of international language content right about that time on various sub-domains on the same server. Its hard to tell from webmaster tools, but it may be that Googlebot is spending more time crawling that and less crawling my existing content.

bigcrags

5+ Year Member



 
Msg#: 4448398 posted 8:13 am on May 4, 2012 (gmt 0)

Thanks for the replies. It seems to be still happening - bizarre. Googlebot is gobbling 1.5-2 Gig of bandwidth per day. Might try limiting their activity - it's not like the site's getting any traffic benefit from Google now anyway.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved