Google Updates and SERP Changes - September 2017

[martinibuster]

---

System: The following 4 messages were cut out of thread at: https://www.webmasterworld.com/google/4860963.htm [webmasterworld.com] by robert_charlton - 3:25 am on Sep 2, 2017 (PDT -8)

---

How do you compare your pages with those in the top 10?

User intent.

[Cralamarre]

It did seem strange to me that so much of my recent traffic increase was from direct traffic, and so little of it was from Google. I guess it makes sense if a lot of these visitors are logged into Google and the referrer is not being passed along. It would also explain (I think) why both my Google and direct traffic dropped by roughly the same amounts when I was hit by Google updates in March and May. I could never understand why a Google update affected my direct traffic.

[sandwich]

Site utterly tanked today and is now ranking/bouncing all over creation. Anyone seeing other weirdness today? Strange bird...

[robert76]

Yes on weirdness. I see some sort of rollback. A page in cache is newer than the page being displayed in search results.

[SnowMan68]

Seeing a lot of movement. Wonder if it's related to the update/pullback over the weekend. Could be in for a big one overnight into tomorrow.

[mosxu]

All I see is zombies behaviour has changed, I think they are trying to look more natural and creating less gaps like we are some sort of stupid human beings...

[Cralamarre]

If I could ask another question regarding Google traffic and Analytics reports... Earlier it was mentioned that when someone visits a non-https site while logged into their Google account, the referrer is not passed along. This causes Google Analytics to report it as direct traffic, not Google traffic, even if they're visiting from a Google search. If that's true, and Google Analytics is reporting an unknown amount of Google traffic as direct traffic, then does that mean non-https sites like mine have no way of accurately knowing how much traffic they receive from Google? Is Google Analytics useless for non-https sites when it comes to reporting traffic sources?

[NickMNS]

Again I'm not 100% certain about this so it would be helpful if someone more knowledgeable could chime in. But, my understanding is that referrer is not included in the http request so it is not that Google Analytics is blocking the information. The information is not passed along at all so you would not even know from your raw server logs.

[aristotle]

Only the newest browsers (and not all of them) follow the "rule" about not revealing the referer when following a link from an https site to an http site. Also, I've never heard that being logged into google has anything to do with it, and doubt that it is true.

[keyplyr]

Only the newest browsers (and not all of them) follow the "rule" about not revealing the referer when following a link from an https site to an http site.

True. It also has to do with how the server is config'd, on the HTTPS site. The fields you see in the access log reports is set up that way either by OS default or by the admin.

I haven't used Google Analytics for years, and many things have changed, but if the browser is not supporting referrer, then it won't be in any report, no matter if you're signed into your Google account or not. If it ain't there, it ain't there :)

[NickMNS]

Thanks Keyplyr and Aristotle
Around here we say: "Je vais me coucher moins niaiseux ce soir"
Translated: "I will go to bed less stupid tonight!"

[Maximum44]

The true referral data have been missing from Google for quite some time. They just give you sample data. I thought that was common understanding as it has been like that for a long time.

[keyplyr]

@Maximum44 - yes that's been the case for several years from Google properties, but not all data comes from Google.

[Shaddows]

It's not so much about being logged in to Google, rather that Google moved it's properties to https.

As others have said, it's not necessarily implemented by the browsers correctly, but the referrer MUST be excluded in a HTTPS to HTTP referral.

The Referer field has the potential to reveal information about the request context or browsing history of the user, which is a privacy concern if the referring resource's identifier reveals personal information (such as an account name) or a resource that is supposed to be confidential (such as behind a firewall or internal to a secured service). Most general-purpose user agents do not send the Referer header field when the referring resource is a local "file" or "data" URI. A user agent MUST NOT send a Referer header field in an unsecured HTTP request if the referring page was received with a secure protocol.
RFC7231 Section 5.5.2 (Standards Track RFC) [tools.ietf.org]
[Emphasis Mine]

[RedBar]

Oh, I haved logged-in this morning to a whole swathe of sites with 0 PVs after 10.5 hours

This is getting more than ridiculous, what the heck is G manipulating now?

[aristotle]

As others have said, it's not necessarily implemented by the browsers correctly, but the referrer MUST be excluded in a HTTPS to HTTP referral.

Shaddows -- Hasn't that specification been around for a good while, yet a lot of people are still using browsers that don't adhere to it.

And as more and more people upgrade their browsers, less and less referal information on visits to http sites will be available. So the problem will only get worse for opwners of those sites.

But for referals from google, you can still find exact information in GSC, at least for the main keywords

[Shaddows]

So the problem will only get worse for opwners of those sites

Indeed. As most high-status sites go https, anyone left on http will start to lose referrer data from a wider set of referrers. This will seriously undermine all analytics packages, including GA.

GSC will not help with non-Google https-sourced traffic.

(Also note that the referring server can set preferences for referral strings, including stripping data. I have absolutely no idea how strictly browsers support those preferences. You can find a horribly-presented explanation here [w3c.github.io])

[Mark_A]

So can I get this right. An https page will not send referrer information to a non https page? Will it send referrer information to a page on a different site that is https? or are we all going to have to get used to a lot less referrer information from now on?

[Shaddows]

An https page will not send referrer information to a non https page?

A compliant browser will not, no.

Will it send referrer information to a page on a different site that is https?

The host site can set preferences which may be supported by a browser. See here:
[w3c.github.io...]

are we all going to have to get used to a lot less referrer information from now on?

"All" - possibly. HTTP-only* - definitely.

*I'm not sure if a link directed at http, which is then 301 to https will retain the referrer. I would assume not- in which case, everyone will lose data on historic links.

[aristotle]

I'm not sure if a link directed at http, which is then 301 to https will retain the referrer. I would assume not

That's a good question -- I hadn't thought of it.

Most likely a browser can be programmed to carry referal data through a 301, but this doesn't mean that it will be done.

[MouseHunter]

Been working on a new site with little authority for a few months now and we were just starting to see some good results in SERPs. Whatever happened last night has destroyed several months of hard work. Look forward to hearing from others with similar experience.

[Cralamarre]

So it seems, based on the discussion, that Google Analytics is less than accurate when it comes to reporting traffic sources. What about overall traffic numbers? Is it less than accurate there as well?

And are there any tools similar to GA that give more accurate results? Or is GA considered "good enough" for the average site?

[Kelowna]

Site utterly tanked today and is now ranking/bouncing all over creation. Anyone seeing other weirdness today? Strange bird...

Google has been de-indexing blog networks early in Sept, you are probably feeling the effect. I personally know of about 300 sites getting the axe from 2 different networks.

The ripple effect will carry on for a while

[renatovieira]

Again I see zombie traffic. Low conversion + high traffic flow.

Someone else?

Tourism, USA.

[Cralamarre]

Other than the panic from waking up to a long list of email notifications this morning from this thread (which turned out to be a discussion about http referrers), I'm not seeing any signs of an update today. Traffic is down slightly this morning compared to yesterday, but nothing to indicate an update. Not yet anyway.

I haven't really felt the effects of any updates since the big one in May. Traffic is still down roughly 18% from before the May update, but at least it has recovered from the terribly slow summer.

[aristotle]

And are there any tools similar to GA that give more accurate results?

GSC provides full and accurate information about referals from google. This is because it gets its information from internal google data bases of search activity. No other tool has access to this information.

Similarly, bing webmaster tools provides full and accurate information about referals from bing

For non-google and non-bing traffic, no tool provides full information.

[Shaddows]

For non-google and non-bing traffic, no tool provides full information.

And, cannot provide full information.

All data is ultimately provided by the browser. If the browser isn't telling, there is nothing any analytics package can do about it.

[Cralamarre]

But what about overall traffic numbers? Is Google Analytics accurately reporting the amount of traffic you receive, even if it can't always tell you where it came from?

[glakes]

But what about overall traffic numbers? Is Google Analytics accurately reporting the amount of traffic you receive, even if it can't always tell you where it came from?

It can't be accurate since so many ad blockers and anti-virus solutions block the GA javascript from firing.

[Cralamarre]

Interesting. So there's really no way for any website to know with 100% accuracy how much traffic it receives? In my case, I'm not sure if I would really benefit from knowing the exact numbers anyway. Google Analytics at least lets me keep track of my site's general performance, which is all I really need.

[Shaddows]

We trialled Piwik, but prefer GA on web analytics.

Piwik can analyse your server logs if you don't want to interrogate them yourself.

Web Analytics of any type run scripts. I block scripts. I could make a strong guess on some others here who also block scripts. Anyone who block scripts (or makes creative use of a hosts file) will defeat analytics programs.

You cannot defeat server logs, in any meaningful sense.

So, GA will not capture everything, but it will capture >99% of real users and exclude most bots. There is no web analytics that can improve that proposition, though others may have a view on execution.

Other than that, you have server logs, which are basically Reality.

Edit - I see glakes was much quicker than me!