Welcome to WebmasterWorld Guest from 54.145.39.186

Message Too Old, No Replies

Massive Crawl by Googlebot

     

kellyman

12:18 am on Mar 28, 2014 (gmt 0)



Has anyone else notice a big crawl by google going on, 3 large sites being fully crawled over the last few days

Good job im on a good fast server

not2easy

3:07 am on Mar 28, 2014 (gmt 0)

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month



They have been hitting hard on 3 of mine. One got so bad I put a 1.25 minute crawl delay. It is like they got off the leash. I'm on a good server, but others are trying to see things too, and they are overdoing it.

ibeau

4:51 am on Mar 28, 2014 (gmt 0)



Yep. For me it went from the largest crawl I've seen over the past 2 years, down to pretty much the lowest I've seen since my site started!

[i.imgur.com...]

Further analysis showed me that Googlebot has only crawled 1 (One) page over the past 24 hours..

kellyman

7:46 am on Mar 28, 2014 (gmt 0)



I have a feeling their about to run a massive update, see the same last year just before penguin refresh

was just wondering if others were seeing the same

Shepherd

9:41 am on Mar 28, 2014 (gmt 0)

Top Contributors Of The Month



Just posted in another thread, Looking at WMT, massive crawl spike on 3/22 - 3/26 (last day of reporting), from 650 pages a day to 13,000 pages a day.

alika

1:25 pm on Mar 28, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



If your site is being hit with massive crawling, does that mean that it looks like your site will be affected by any new update they run?

So if your site is not seeing any massive uptick in crawling, does that mean your site is not likely to get affected by this soon-to-come update?

I guess my question is: when you are seeing massive uptick in crawling, is that a good or a bad thing?

[edited by: alika at 1:30 pm (utc) on Mar 28, 2014]

kellyman

1:35 pm on Mar 28, 2014 (gmt 0)



Think they need to crawl to find out, I saw this kind of crawl just before penguin refresh in October

Shepherd

2:12 pm on Mar 28, 2014 (gmt 0)

Top Contributors Of The Month



Based on the history of penguin we are due for an update soon, could be what we are seeing, the precursor.

EditorialGuy

3:23 pm on Mar 28, 2014 (gmt 0)

WebmasterWorld Senior Member Top Contributors Of The Month



Googlebot has been in maintenance mode on my sites (nothing even remotely deep since early March).

nomis5

10:36 pm on Mar 28, 2014 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



My largest site has shown a small decrease in crawl rate over the last few days. Two of my smaller sites have shown a very definite large increase in crawl rate over that time. Not the highest but very near.

I have not the slightest clue what use that information is to me or what use I can make of it. Seems like a case of whatever will be, will be.

ibeau

11:02 pm on Mar 28, 2014 (gmt 0)



Well after the massive crawl GoogleBot has only crawled 108 pages of my site within the last 24 hours. (BingBot was up around 13,000).

Weirdly though my impressions have gone through the roof, while clicks have stayed the same. Must be a bug -> [i.imgur.com...]
 

Featured Threads

Hot Threads This Week

Hot Threads This Month