Welcome to WebmasterWorld Guest from 3.227.233.6

Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Googlebot VERY Active - Good News?

     
3:51 pm on Nov 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:May 29, 2003
posts:890
votes: 55


Googlebot has been very consistent, crawling about 200 of my ~400 pages daily.
Yesterday, they crawled nearly 1,100 pages (or every page about 3 times).

I am hoping that this indicates my pages will all soon be HTTPS.
I've been around 60% HTTPS for months, after having not redirected properly, until a week ago.

Or, have others also been getting super-crawled as well?
Maybe for the new mobile-only index?

Thoughts?
.
6:09 pm on Nov 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Sept 14, 2011
posts:1045
votes: 132


I checked a few sites no unusual Google activity in last 24 hours, so maybe its related to the correction you made on redirects signaling a deep crawl.
6:51 pm on Nov 12, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:May 29, 2003
posts:890
votes: 55


Thanks, seoskunk.
Hopefully, your suggestion is correct.

Going https ain't as easy as is repeatedly suggested, unless you have done it a bunch of times.
But for me, a first-timer, with a 14-year-old site, it has been a living nightmare of 10 hours a day, 6 days a week, for months.
So much junk had accumulated, things like a single period with an http link, stuff like that.
About 69% of pages converted now, still waiting for the rest.
I hope this aggressive crawling activity is indicative of an upcoming jump forward.
.
7:28 pm on Nov 12, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15754
votes: 827


When I moved my personal site to HTTPS, Google crawled it exactly as if it were a new site. That is, it didn't just follow redirects as and when it met them in the course of ordinary requests; once it had discovered the “new” front page, it did a systematic spidering of the entire site. So for a short time you will see a real spike in Googlebot requests.

You may also see a doubling in requests for the front page, as it continues month after month confirming that everything is getting redirected to HTTPS.
8:22 am on Nov 13, 2017 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Aug 11, 2008
posts:1717
votes: 263


When I moved my personal site to HTTPS, Google crawled it exactly as if it were a new site.

I concur. As soon as Google discovered our https redirect, we were crawled top to bottom.

Very shortly afterwards, all our listings were in https.
3:23 pm on Nov 13, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:May 29, 2003
posts:890
votes: 55


That was it.
I went from 69% to 90% HTTPS indexed this morning, in my GSC, under Crawl - Sitemaps - Indexed.
In GSC, under Google Index - Index Status, it hasn't changed yet, but I expect it will shortly.

So, just a few days, AFTER the deep crawl, AFTER I got my redirection correct.
I hope for smooth sailing, going forward.
Now, I can finally go back to creating content.
.
3:32 pm on Nov 13, 2017 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member redbar is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Oct 14, 2013
posts:3229
votes: 496


Yep, I've seen some good improvements and updates the last few day with cache dates even showing for Saturday (11th) on newly ranked pages on G.com's first page.
10:32 pm on Nov 16, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:May 29, 2003
posts:890
votes: 55


This morning, I went to 94% HTTPS.
So, yes, the DEEP crawl seems to be indicative of imminent further HTTPS indexing.
A weight off my shoulders.
PLEASE, check that you are redirecting properly, or it will take MONTHS to get HTTPS indexed.
Cheers.
.
2:44 am on Nov 19, 2017 (gmt 0)

New User

joined:Nov 18, 2017
posts:1
votes: 0


It is a good news for all us bloggers that our posts are now indexing very quickly and Google now acting fast so one of my post also index now which was waiting from some time.
4:39 pm on Nov 22, 2017 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:May 29, 2003
posts:890
votes: 55


This morning, I went to 99% HTTPS indexed.
The deep crawl is over, my pages are converted, and I am a happy camper.
Time to get to work on new content. And chase after other issues.
.
11:00 pm on Nov 29, 2017 (gmt 0)

New User from VN 

joined:Sept 26, 2017
posts:1
votes: 0


I think this is a good signal for Seoer but It is a bad news for Spam.
12:38 am on Nov 30, 2017 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15754
votes: 827


I just took a closer look at GSC index data for my personal site, which went https in May because the browser kept yapping about piwik login. It's easy to check, as it's got a whopping total of something like 20 public pages. Interestingly but not especially surprisingly, the http and https charts aren't parallel. The https side leapt straight from 0 to 16 between one week and the next, while http clambered slowly and hiccupily downward over the course of 2-3 months. Since late July, they have consistently claimed to have three http pages indexed, dropping down to 2 at one point and then climbing back up again. I've got a nasty feeling this is a reference to my boilerplate pages--contact, legal and so on--which are not supposed to be crawled at all. (This is one reason I eventually exempted robots.txt from the https redirect.)