Welcome to WebmasterWorld Guest from 54.158.195.221

Forum Moderators: Robert Charlton & andy langton & goodroi

Message Too Old, No Replies

Massive Crawl by Googlebot

     
12:18 am on Mar 28, 2014 (gmt 0)

Full Member

5+ Year Member

joined:Oct 11, 2010
posts: 227
votes: 0


Has anyone else notice a big crawl by google going on, 3 large sites being fully crawled over the last few days

Good job im on a good fast server
3:07 am on Mar 28, 2014 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:3405
votes: 172


They have been hitting hard on 3 of mine. One got so bad I put a 1.25 minute crawl delay. It is like they got off the leash. I'm on a good server, but others are trying to see things too, and they are overdoing it.
4:51 am on Mar 28, 2014 (gmt 0)

Junior Member

5+ Year Member

joined:Apr 25, 2012
posts: 53
votes: 0


Yep. For me it went from the largest crawl I've seen over the past 2 years, down to pretty much the lowest I've seen since my site started!

[i.imgur.com...]

Further analysis showed me that Googlebot has only crawled 1 (One) page over the past 24 hours..
7:46 am on Mar 28, 2014 (gmt 0)

Full Member

5+ Year Member

joined:Oct 11, 2010
posts: 227
votes: 0


I have a feeling their about to run a massive update, see the same last year just before penguin refresh

was just wondering if others were seeing the same
9:41 am on Mar 28, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Oct 5, 2012
posts:752
votes: 84


Just posted in another thread, Looking at WMT, massive crawl spike on 3/22 - 3/26 (last day of reporting), from 650 pages a day to 13,000 pages a day.
1:25 pm on Mar 28, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 27, 2003
posts:1597
votes: 0


If your site is being hit with massive crawling, does that mean that it looks like your site will be affected by any new update they run?

So if your site is not seeing any massive uptick in crawling, does that mean your site is not likely to get affected by this soon-to-come update?

I guess my question is: when you are seeing massive uptick in crawling, is that a good or a bad thing?

[edited by: alika at 1:30 pm (utc) on Mar 28, 2014]

1:35 pm on Mar 28, 2014 (gmt 0)

Full Member

5+ Year Member

joined:Oct 11, 2010
posts: 227
votes: 0


Think they need to crawl to find out, I saw this kind of crawl just before penguin refresh in October
2:12 pm on Mar 28, 2014 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member Top Contributors Of The Month

joined:Oct 5, 2012
posts:752
votes: 84


Based on the history of penguin we are due for an update soon, could be what we are seeing, the precursor.
3:23 pm on Mar 28, 2014 (gmt 0)

Senior Member

WebmasterWorld Senior Member editorialguy is a WebmasterWorld Top Contributor of All Time Top Contributors Of The Month

joined:June 28, 2013
posts:2972
votes: 521


Googlebot has been in maintenance mode on my sites (nothing even remotely deep since early March).
10:36 pm on Mar 28, 2014 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Apr 29, 2005
posts:2001
votes: 75


My largest site has shown a small decrease in crawl rate over the last few days. Two of my smaller sites have shown a very definite large increase in crawl rate over that time. Not the highest but very near.

I have not the slightest clue what use that information is to me or what use I can make of it. Seems like a case of whatever will be, will be.
11:02 pm on Mar 28, 2014 (gmt 0)

Junior Member

5+ Year Member

joined:Apr 25, 2012
posts: 53
votes: 0


Well after the massive crawl GoogleBot has only crawled 108 pages of my site within the last 24 hours. (BingBot was up around 13,000).

Weirdly though my impressions have gone through the roof, while clicks have stayed the same. Must be a bug -> [i.imgur.com...]