Welcome to WebmasterWorld Guest from 34.204.194.190

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Big spike in Google Crawl

     
10:26 am on Jun 9, 2015 (gmt 0)

New User

joined:May 6, 2015
posts: 9
votes: 0


Hey guys,

It seems yesterday that one of my clients saw a huge spike in pages crawled by Google - something like a 6-fold increase, and wondered if anyone else has seen the same? I have heard that this is quite common before Panda / Penguin in the past and just wondered if anyone else has seen the same?

Cheers,

Andy
11:54 am on June 9, 2015 (gmt 0)

New User

joined:May 6, 2015
posts: 9
votes: 0


Sorry for the poor wording - tried to edit but was too late.
2:54 pm on June 10, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member editorialguy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 28, 2013
posts:3466
votes: 778


In my experience, an unusually deep crawl used to mean that something was about to happen: If I saw a big spike, I could be fairly sure that a major update would occur within a few days.

That doesn't seem to be true anymore--at least for us. Over the last year, we've had a number of unusually deep crawls (maybe one every 6 or 8 weeks) with no obvious changes in our search rankings or Google organic traffic afterwards.
2:49 am on June 12, 2015 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 16, 2005
posts:100
votes: 0


I saw the same. A ginormous spike 6/8-6/9 like I don't remember ever seeing.
7:15 am on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member editorialguy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 28, 2013
posts:3466
votes: 778


We had a pretty big spike on June 9, too, but it was only about 3/4 as big as our peak crawl in April (which didn't lead to any obvious ups or downs in Google traffic).
9:29 am on June 12, 2015 (gmt 0)

Junior Member

10+ Year Member

joined:June 13, 2006
posts:188
votes: 0


We have been told to expect a Panda update,(about now to the next couple weeks) which is more about Google bringing their index up to date rather than a new algo. So this all makes sense to me.
9:33 am on June 12, 2015 (gmt 0)

New User

joined:May 6, 2015
posts: 9
votes: 0


I think that pretty much sums it up Flanok.
9:51 am on June 12, 2015 (gmt 0)

Full Member

10+ Year Member Top Contributors Of The Month

joined:June 3, 2005
posts: 298
votes: 12


@AndyD, @Purplekitty and @EditorialGuy, it would be handy to know if these are Panda hit sites you are seeing a spike on. And if images are being spidered (see the other thread I started). I have had my spike on my Panda effected site.

I advise anyone removing/adding Panda related content after the spike do a "fetch as google" in WMT this month,
10:02 am on June 12, 2015 (gmt 0)

New User

joined:May 6, 2015
posts: 9
votes: 0


The site was hit in the past Johan, but made a better-than-partial recovery some time back.

And yes, Fetch as Google would be a good move.
1:01 pm on June 12, 2015 (gmt 0)

Preferred Member from ES 

10+ Year Member Top Contributors Of The Month

joined:Mar 4, 2003
posts: 491
votes: 6


I saw a very large spike on 24th May on a site that was hit by panda. My other sites, which were not hit by panda, have not shown any crawling spikes.
1:07 pm on June 12, 2015 (gmt 0)

Full Member

10+ Year Member Top Contributors Of The Month

joined:June 3, 2005
posts: 298
votes: 12


My Panda affected site spike was on 11th May. Mainly images though!
2:36 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member editorialguy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 28, 2013
posts:3466
votes: 778


Johan007: We were affected by Panda early on (not so much "hit" as nudged back a few spots by pages from "big brand" sites, which had an obvious effect on traffic), However, our Google traffic jumped 250 to 300 percent after last year's Panda update, so I don't think we could be termed a "Panda site" these days (except in a good way).

Flanok does make a good point, though.
4:16 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:3652
votes: 369


Well does a site that "fully recovers" from Panda start over with a totally clean slate? Or does it still carry the stigma and be stained forever after?

If it starts over with a clean slate, then a "deep crawl" would be irrelevant as a predictor of a looming Panda refresh.
4:36 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member editorialguy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 28, 2013
posts:3466
votes: 778


Well does a site that "fully recovers" from Panda start over with a totally clean slate? Or does it still carry the stigma and be stained forever after?

Panda isn't a penalty (though it may look like one if you're affected), so there shouldn't be any "stigma."
5:44 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:3652
votes: 369


Quote:
Stigma: an ancient name for marks that were pricked or branded onto the bodies of slaves and soldiers for identification purposes. The word was also applied to religious symbols tattooed onto members of pagan religious cults to show their devotion to particular gods and goddesses.

So it's a "mark". In other words, it could be an indicator of a site that is (or was) affected by Panda. Google could have it in their permanent record of a site's history.
6:49 pm on June 12, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15809
votes: 847


So it's a "mark".

I'm all for etymological digressions, but I honestly don't think it's useful here. The reference is to the modern usage: the word whose plural is stigmas, not stigmata.
7:14 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member aristotle is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Aug 4, 2008
posts:3652
votes: 369


Well Lucy my questions concerned how Google treats the site. Is the slate wiped completely clean? Or does the site continue to carry some kind of indication (in Google's records) that it was once affected by Panda, and which might cause it to be crawled more deeply in the weeks before a Panda refresh, which some of the earlier comments appeared to assume.
7:20 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member editorialguy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 28, 2013
posts:3466
votes: 778


'm all for etymological digressions, but I honestly don't think it's useful here. The reference is to the modern usage: the word whose plural is stigmas, not stigmata.

To paraphrase Shakespeare, "If Panda pricks us, do we not bleed?"
7:29 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Sept 14, 2011
posts:1045
votes: 132


Shakespeare was indeed a clever chap, I never knew he was an seo as well...
8:23 pm on June 12, 2015 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator robert_charlton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2000
posts:12362
votes: 403


Just to get this thread back on topic, are these stigmata attributable to the spike in crawling?
8:28 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Sept 14, 2011
posts:1045
votes: 132


Stigmata .... now the thread is turning all Christian, that's the marks of the cross
8:36 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member editorialguy is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:June 28, 2013
posts:3466
votes: 778


Instead of crying "Mamma Mia!" when Googlebot goes crazy, I guess I'll have to exclaim "Padre Pio!"
8:41 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Sept 14, 2011
posts:1045
votes: 132


LOL Yeah instead of hiring an seo to recover your site you'll have to get an exorcist....
9:04 pm on June 12, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:June 6, 2006
posts:1191
votes: 41


It used to be said that the only way to get a site listed by Yahoo was to scatter fresh chicken blood at midnight. I tried it but it didn't work.

However I have seen no sign whatsoever of a Googlebot spike so there'll be no sabbat this weekend after all.
3:08 am on June 13, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15809
votes: 847


But seriously Andy... Are they requesting current, active pages? I ask because not long ago I got a blizzard of google requests for URLs that have been gone (301 or 410) for up to four years, leading me to suspect some kind of housecleaning at the googleplex.

are these stigmata attributable to the spike

I will never, ever forgive myself for not thinking of this first.
9:26 am on June 13, 2015 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 11, 2008
posts: 192
votes: 0


Our site got hit by Panda in October 2014. After making lots of changes we are hoping to make a recovery with the next update.

The last big spike in crawling we have seen was on April 6.

Our site has approximately 600 pages but I notice that Webmaster Tools report an average of 3,856 pages crawled per day. Is this normal?
1:28 pm on June 13, 2015 (gmt 0)

Full Member

10+ Year Member Top Contributors Of The Month

joined:June 3, 2005
posts: 298
votes: 12


Our site has approximately 600 pages but I notice that Webmaster Tools report an average of 3,856 pages crawled per day. Is this normal?
Not ideal is it. Perhaps means nothing if you ask John Mueller but under Panda I would not risk this. Run site query in Google and add canonical tags to article pages in the first instance and random deleted pages need to be 410 or 301 to the correct location. After a couple of months then consider Robots.txt with wildcards to clean up chunks of URL's that have a pattern.

My robots have these lines to remove WordPress stuff and pages with funny query strings added:

User-agent: *
Disallow: /wp-admin/
Disallow: /page/ # remove pagination on root
Disallow: /tag/*/page/* # prevents robots from indexing tag pagination
Disallow: /category/*/page/*?* # prevents robots from indexing query strings on pagination
2:11 pm on June 13, 2015 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 11, 2008
posts: 192
votes: 0


We have tag pages etc set to no-index so I don't think this would be a problem. Or should we disallow these in robots.txt instead of no-index?

We have canonicals on all pages as well.

In Google Webmaster Tools we only have 600 pages indexed so Google doesn't seem to be indexing pages it shouldn't. Just seems to be crawling a lot.
3:52 pm on June 13, 2015 (gmt 0)

Full Member

10+ Year Member Top Contributors Of The Month

joined:June 3, 2005
posts: 298
votes: 12


We have tag pages etc set to no-index so I don't think this would be a problem. Or should we disallow these in robots.txt instead of no-index?
That is right. You need to be given access without blocking via robots to be able to know if to remove them via 410 or "noindex" otherwise manually removing 3000 urls via WMT (+ robots) is impossible.

In Google Webmaster Tools we only have 600 pages indexed so Google doesn't seem to be indexing pages it shouldn't.
That is normal I have far less shown in WMT.

Do a "site:domain.com" and see if you can mass exclude with wildcards perhaps using htaccess to 410 in the first instance then a month down the line robots.txt.
8:35 pm on June 13, 2015 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 16, 2005
posts:100
votes: 0


I was hit by something in 2012. I have no idea what, but suspect panda because that's what people told me it probably was. Whatever it was, it affected my traffic in a huge way and there was no way for me to know what the problem was because I wasn't doing anything against Google guidelines. I suddenly had recovery in July 2013. I haven't had any issues since that recovery and just continue to do my thing as I've done since I started the websites.

I just went back and looked at the spike from a few days ago. It is twice that of "normal" spiking that I see every month or so, and more than 4x that of the average.
This 31 message thread spans 2 pages: 31
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members