Big spike in Google Crawl

[AndyD]

Hey guys,

It seems yesterday that one of my clients saw a huge spike in pages crawled by Google - something like a 6-fold increase, and wondered if anyone else has seen the same? I have heard that this is quite common before Panda / Penguin in the past and just wondered if anyone else has seen the same?

Cheers,

Andy

[AndyD]

Sorry for the poor wording - tried to edit but was too late.

[EditorialGuy]

In my experience, an unusually deep crawl used to mean that something was about to happen: If I saw a big spike, I could be fairly sure that a major update would occur within a few days.

That doesn't seem to be true anymore--at least for us. Over the last year, we've had a number of unusually deep crawls (maybe one every 6 or 8 weeks) with no obvious changes in our search rankings or Google organic traffic afterwards.

[purplekitty]

I saw the same. A ginormous spike 6/8-6/9 like I don't remember ever seeing.

[EditorialGuy]

We had a pretty big spike on June 9, too, but it was only about 3/4 as big as our peak crawl in April (which didn't lead to any obvious ups or downs in Google traffic).

[flanok]

We have been told to expect a Panda update,(about now to the next couple weeks) which is more about Google bringing their index up to date rather than a new algo. So this all makes sense to me.

[AndyD]

I think that pretty much sums it up Flanok.

[Johan007]

@AndyD, @Purplekitty and @EditorialGuy, it would be handy to know if these are Panda hit sites you are seeing a spike on. And if images are being spidered (see the other thread I started). I have had my spike on my Panda effected site.

I advise anyone removing/adding Panda related content after the spike do a "fetch as google" in WMT this month,

[AndyD]

The site was hit in the past Johan, but made a better-than-partial recovery some time back.

And yes, Fetch as Google would be a good move.

[denisl]

I saw a very large spike on 24th May on a site that was hit by panda. My other sites, which were not hit by panda, have not shown any crawling spikes.

[Johan007]

My Panda affected site spike was on 11th May. Mainly images though!

[EditorialGuy]

Johan007: We were affected by Panda early on (not so much "hit" as nudged back a few spots by pages from "big brand" sites, which had an obvious effect on traffic), However, our Google traffic jumped 250 to 300 percent after last year's Panda update, so I don't think we could be termed a "Panda site" these days (except in a good way).

Flanok does make a good point, though.

[aristotle]

Well does a site that "fully recovers" from Panda start over with a totally clean slate? Or does it still carry the stigma and be stained forever after?

If it starts over with a clean slate, then a "deep crawl" would be irrelevant as a predictor of a looming Panda refresh.

[EditorialGuy]

Well does a site that "fully recovers" from Panda start over with a totally clean slate? Or does it still carry the stigma and be stained forever after?

Panda isn't a penalty (though it may look like one if you're affected), so there shouldn't be any "stigma."

[aristotle]

Quote:
Stigma: an ancient name for marks that were pricked or branded onto the bodies of slaves and soldiers for identification purposes. The word was also applied to religious symbols tattooed onto members of pagan religious cults to show their devotion to particular gods and goddesses.

So it's a "mark". In other words, it could be an indicator of a site that is (or was) affected by Panda. Google could have it in their permanent record of a site's history.

[lucy24]

So it's a "mark".

I'm all for etymological digressions, but I honestly don't think it's useful here. The reference is to the modern usage: the word whose plural is stigmas, not stigmata.

[aristotle]

Well Lucy my questions concerned how Google treats the site. Is the slate wiped completely clean? Or does the site continue to carry some kind of indication (in Google's records) that it was once affected by Panda, and which might cause it to be crawled more deeply in the weeks before a Panda refresh, which some of the earlier comments appeared to assume.

[EditorialGuy]

'm all for etymological digressions, but I honestly don't think it's useful here. The reference is to the modern usage: the word whose plural is stigmas, not stigmata.

To paraphrase Shakespeare, "If Panda pricks us, do we not bleed?"

[seoskunk]

Shakespeare was indeed a clever chap, I never knew he was an seo as well...

[Robert Charlton]

Just to get this thread back on topic, are these stigmata attributable to the spike in crawling?

[seoskunk]

Stigmata .... now the thread is turning all Christian, that's the marks of the cross

[EditorialGuy]

Instead of crying "Mamma Mia!" when Googlebot goes crazy, I guess I'll have to exclaim "Padre Pio!"

[seoskunk]

LOL Yeah instead of hiring an seo to recover your site you'll have to get an exorcist....

[superclown2]

It used to be said that the only way to get a site listed by Yahoo was to scatter fresh chicken blood at midnight. I tried it but it didn't work.

However I have seen no sign whatsoever of a Googlebot spike so there'll be no sabbat this weekend after all.

[lucy24]

But seriously Andy... Are they requesting current, active pages? I ask because not long ago I got a blizzard of google requests for URLs that have been gone (301 or 410) for up to four years, leading me to suspect some kind of housecleaning at the googleplex.

are these stigmata attributable to the spike

I will never, ever forgive myself for not thinking of this first.

[almo136]

Our site got hit by Panda in October 2014. After making lots of changes we are hoping to make a recovery with the next update.

The last big spike in crawling we have seen was on April 6.

Our site has approximately 600 pages but I notice that Webmaster Tools report an average of 3,856 pages crawled per day. Is this normal?

[Johan007]

Our site has approximately 600 pages but I notice that Webmaster Tools report an average of 3,856 pages crawled per day. Is this normal?

Not ideal is it. Perhaps means nothing if you ask John Mueller but under Panda I would not risk this. Run site query in Google and add canonical tags to article pages in the first instance and random deleted pages need to be 410 or 301 to the correct location. After a couple of months then consider Robots.txt with wildcards to clean up chunks of URL's that have a pattern.

My robots have these lines to remove WordPress stuff and pages with funny query strings added:

User-agent: *
Disallow: /wp-admin/
Disallow: /page/ # remove pagination on root
Disallow: /tag/*/page/* # prevents robots from indexing tag pagination
Disallow: /category/*/page/*?* # prevents robots from indexing query strings on pagination

[almo136]

We have tag pages etc set to no-index so I don't think this would be a problem. Or should we disallow these in robots.txt instead of no-index?

We have canonicals on all pages as well.

In Google Webmaster Tools we only have 600 pages indexed so Google doesn't seem to be indexing pages it shouldn't. Just seems to be crawling a lot.

[Johan007]

We have tag pages etc set to no-index so I don't think this would be a problem. Or should we disallow these in robots.txt instead of no-index?

That is right. You need to be given access without blocking via robots to be able to know if to remove them via 410 or "noindex" otherwise manually removing 3000 urls via WMT (+ robots) is impossible.

In Google Webmaster Tools we only have 600 pages indexed so Google doesn't seem to be indexing pages it shouldn't.

That is normal I have far less shown in WMT.

Do a "site:domain.com" and see if you can mass exclude with wildcards perhaps using htaccess to 410 in the first instance then a month down the line robots.txt.

[purplekitty]

I was hit by something in 2012. I have no idea what, but suspect panda because that's what people told me it probably was. Whatever it was, it affected my traffic in a huge way and there was no way for me to know what the problem was because I wasn't doing anything against Google guidelines. I suddenly had recovery in July 2013. I haven't had any issues since that recovery and just continue to do my thing as I've done since I started the websites.

I just went back and looked at the spike from a few days ago. It is twice that of "normal" spiking that I see every month or so, and more than 4x that of the average.