Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Googlebot VERY Active - Good News?

         

Sally Stitts

3:51 pm on Nov 12, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Googlebot has been very consistent, crawling about 200 of my ~400 pages daily.
Yesterday, they crawled nearly 1,100 pages (or every page about 3 times).

I am hoping that this indicates my pages will all soon be HTTPS.
I've been around 60% HTTPS for months, after having not redirected properly, until a week ago.

Or, have others also been getting super-crawled as well?
Maybe for the new mobile-only index?

Thoughts?
.

seoskunk

6:09 pm on Nov 12, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I checked a few sites no unusual Google activity in last 24 hours, so maybe its related to the correction you made on redirects signaling a deep crawl.

Sally Stitts

6:51 pm on Nov 12, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Thanks, seoskunk.
Hopefully, your suggestion is correct.

Going https ain't as easy as is repeatedly suggested, unless you have done it a bunch of times.
But for me, a first-timer, with a 14-year-old site, it has been a living nightmare of 10 hours a day, 6 days a week, for months.
So much junk had accumulated, things like a single period with an http link, stuff like that.
About 69% of pages converted now, still waiting for the rest.
I hope this aggressive crawling activity is indicative of an upcoming jump forward.
.

lucy24

7:28 pm on Nov 12, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



When I moved my personal site to HTTPS, Google crawled it exactly as if it were a new site. That is, it didn't just follow redirects as and when it met them in the course of ordinary requests; once it had discovered the “new” front page, it did a systematic spidering of the entire site. So for a short time you will see a real spike in Googlebot requests.

You may also see a doubling in requests for the front page, as it continues month after month confirming that everything is getting redirected to HTTPS.

Shaddows

8:22 am on Nov 13, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



When I moved my personal site to HTTPS, Google crawled it exactly as if it were a new site.

I concur. As soon as Google discovered our https redirect, we were crawled top to bottom.

Very shortly afterwards, all our listings were in https.

Sally Stitts

3:23 pm on Nov 13, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



That was it.
I went from 69% to 90% HTTPS indexed this morning, in my GSC, under Crawl - Sitemaps - Indexed.
In GSC, under Google Index - Index Status, it hasn't changed yet, but I expect it will shortly.

So, just a few days, AFTER the deep crawl, AFTER I got my redirection correct.
I hope for smooth sailing, going forward.
Now, I can finally go back to creating content.
.

RedBar

3:32 pm on Nov 13, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Yep, I've seen some good improvements and updates the last few day with cache dates even showing for Saturday (11th) on newly ranked pages on G.com's first page.

Sally Stitts

10:32 pm on Nov 16, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



This morning, I went to 94% HTTPS.
So, yes, the DEEP crawl seems to be indicative of imminent further HTTPS indexing.
A weight off my shoulders.
PLEASE, check that you are redirecting properly, or it will take MONTHS to get HTTPS indexed.
Cheers.
.

Shure1975

2:44 am on Nov 19, 2017 (gmt 0)

5+ Year Member



It is a good news for all us bloggers that our posts are now indexing very quickly and Google now acting fast so one of my post also index now which was waiting from some time.

Sally Stitts

4:39 pm on Nov 22, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



This morning, I went to 99% HTTPS indexed.
The deep crawl is over, my pages are converted, and I am a happy camper.
Time to get to work on new content. And chase after other issues.
.

MoiDichVuSeo

11:00 pm on Nov 29, 2017 (gmt 0)

5+ Year Member



I think this is a good signal for Seoer but It is a bad news for Spam.

lucy24

12:38 am on Nov 30, 2017 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



I just took a closer look at GSC index data for my personal site, which went https in May because the browser kept yapping about piwik login. It's easy to check, as it's got a whopping total of something like 20 public pages. Interestingly but not especially surprisingly, the http and https charts aren't parallel. The https side leapt straight from 0 to 16 between one week and the next, while http clambered slowly and hiccupily downward over the course of 2-3 months. Since late July, they have consistently claimed to have three http pages indexed, dropping down to 2 at one point and then climbing back up again. I've got a nasty feeling this is a reference to my boilerplate pages--contact, legal and so on--which are not supposed to be crawled at all. (This is one reason I eventually exempted robots.txt from the https redirect.)