Google to stop giving referral data - by moving to SSL

[Donna]

[googleblog.blogspot.com ]

Is it me or does it means that we no longer will be able to use our server based tracking tools to get google referral data from search ? GA love anyone ?!?! Please correct me that I am wrong.

When you search from [google.com,...] websites you visit from our organic search listings will still know that you came from Google, but won't receive information about each individual query.

From the look of it for the past couple months Google have officialy declared war on all webmasters.

[J_RaD]

Yes the data is available for adword users but it is their property and they can choose to give this useful information to only those who pay them...

so you are saying organic traffic should be dammed and the entire front page of goog should be 1-100 of the best ads? killer!

I don't see a problem in this as Google has long made it clear that users come first for them and not the webmasters.

kool aid mustache? The only thing they've made clear is that googs bottom line comes 1st, they use the "users" word as a silly hide behind tactic. They then use the webmasters as tools to gain more data so they can then shaft the webmasters making goog more goog feeding their bottom line Ehem i mean making searches better for the "user".

t also helps them keep away this data from their bitter rivals Microsoft as googlers weren't happy at what they were doing. "We track data on all websites and how dare you track data on our property through users of your browser?"

right because chrome doesn't track a single thing you do.

the jig is up....goog is just an enormous personal data sucking black hole that wants nothing more then the complete and total control of the net....and that means getting rid of ANYTHING they do not control, IE webmasters that run websites. Every single move they do is to erase something and replace it with "goog this"

[robdwoods]

This pretty much screws Google's advertiser competitors and SEOs. Two groups Google won't care much about upsetting.

[Spiekerooger]

Hi,

I just ran a test to see if referrer data still includes the q param value if the target is ssl secure.

The answer is: no. The search term data is not included in the referrer even if the target is ssl secure and also listed as ssl secure in googles serps.

This was tested against [google.com...]

It problably was (or still is) different on [encrypted.google.com....] In 2010, you would receive the q param value if your site was ssl secure but not if your site was not ssl secure if the blog posts about this back from 2010 and the posts here at webmasterworld are to be believed (I did not test it myself back in 2010).

But using ssl yourself won't help you anymore in extracting the keywords searched for at google with the implementation of ssl at google now. They deliberately strip the search term in the referrer. The referrer still includes google, the cd param value (serp position), the url param value etc. Only the q param value is stripped (e.g. "&q=&").

For anyone thinking that this is forcing webmasters into using GA: No, GA also has no information about the q param value as well, so using GA because of this is useless. (I also never recommend using GA at all...)

They just told in their blog posts that the top search terms over the last 30 days as stated in the webmaster tools will be included in GA. Not the real data that could be obtained by a referrer incl. the q param value.

(q param value is used in this post as a placeholder for q param, og param etc., any params that normally had the search term as a value)

[Leosghost]

Thanks ..that is the impression I had from their "explanation" that they were stripping the q param etc...quite what they think the cd param etc is worth without it is beyond me..and likewise.. I don't use G analytics..they have sufficient data IMO as it is..

Until now..with this move of theirs ..I could get all the data myself .. without Google getting details of who went where after they arrived on a page...Not happy about this move at all >:((

[smallcompany]

IMO, there is no reason why a website owner would not be entitled into knowing which keyword has brought the buyer onto her/his website.

Privacy has nothing to do with it. The seller already knows the name and address and credit card info from the purchaser.
If the purchase has not been made, there are no names or any other personal information anyway.

When people get greeted in the store they're often asked about if they were looking for something in particular. Then, the potential buyer could say "just browsing" or "pants"

Ah, anyway... Google... They have too much freedom compared to how much of everything they actually control. Is that called a monopoly?

[J_RaD]

They deliberately strip the search term in the referrer.

grr SOB!

[rlange]

indyank wrote:
I don't see a problem in this as Google has long made it clear that users come first for them and not the webmasters.Why should they give out this data for free and let the webmasters game them?

It also helps them keep away this data from their bitter rivals Microsoft as googlers weren't happy at what they were doing. "We track data on all websites and how dare you track data on our property through users of your browser?"

Let's not go cartwheeling off into corporate conspiracy land here, folks...

Keep in mind that, once rolled out, this will only affect organic traffic from those searchers who are logged in. Yes, even if it effects less than 10% of users, it's not necessarily non-trivial, but it's also not complete silence.

Google's turning off one of ten lights in your room. Enough, maybe, to be noticeable, but not enough to throw you into complete darkness.

Don't get carried away.

Spiekerooger wrote:
They deliberately strip the search term in the referrer. The referrer still includes google, the cd param value (serp position), the url param value etc. Only the q param value is stripped (e.g. "&q=&").

The Referer header is set by the browser, not the referring page, so I'm curious as to how they're accomplishing this...

Edit: Heh! Answered below as I was updating this post with the question.

--
Ryan

[edited by: rlange at 7:05 pm (utc) on Oct 19, 2011]

[Spiekerooger]

Just to understand how they delete the q param value:

When you click on a result, the link target is e.g.

[google.com...]

(***=some encryted information as sessions,etc., therefor masked by me)

this page will direct you (via javascript) to the target displayed in the serps. As you see, the q param value is deleted here already, therefor your browser (which is acutally sending the referrer header used by any kind of analytics tools) sends this as a referrer and you have no chance to deduct the search term.

[edited by: Spiekerooger at 7:07 pm (utc) on Oct 19, 2011]

[Spiekerooger]

Keep in mind that, once rolled out, this will only affect organic traffic from those searchers who are logged in. Yes, even if it effects less than 10% of users, it's not necessarily non-trivial, but it's also not complete silence.

I do believe that google is following the ssl everywhere doctrine. So this is probably just affecting logged in users now, but will be mainstream soon enough.

I do think that ssl everywhere is a good idea if the ssl system as it is now will be fixed against its major flaws as shown by ssl failures at comodo or diginotar, e.g.

But I do not follow in Googles idea of deleting information about search terms as these are still obtained by Google itself.

So it's not "Don't be evil" at Google, it's "All your data belong to us and ONLY to us".

[rlange]

Spiekerooger wrote:
As you see, the q param value is deleted here already, therefor your browser (which is acutally sending the referrer header used by any kind of analytics tools) sends this as a referrer and you have no chance to deduct the search term.

What the...? If that's true, they could do this without SSL.

--
Ryan

[Spiekerooger]

That's the way google handles their serps with ssl, yes. They do this in the normal non-sll serps as well but they do not delete the q param value there (but could be doing it as well, for sure).

So what's the best answer to this?

Starting a campaign that javascript and meta redirects are evil ;-) If you do switch off javascript and meta redirects, the href target in the serps is the actual target, not a redirect. And then the browser sends all the data in the referrer header (if not deleted/changed by browser settings).

[Leosghost]

What the...? If that's true, they could do this without SSL.

Yes ..but by doing it when playing with SSL ..they get to be able to claim it is for "privacy and security reasons"..for "the user" ( bless them )..and the stripping of the query term ( the real reason for it all ) slides by / goes over the heads / is not understood, by the "non tech people"..or the "user" will fall for it and say yes "they do not want the nasty website that is at the end of the query chain spying on them"..thankyou Google for protecting us" they will think..

And Google speak to them of mysterious things such as man in the middle attacks and hint vaguely at credit card type security and thus the "security theatre" will work..and as Spiekerooger says they'll eventually expand it to "not signed in" ..and "opt out"..like they did with "instant" etc..

[Leosghost]

Starting a campaign that javascript and meta redirects are evil ;-) If you do switch off javascript and meta redirects, the href target in the serps is the actual target, not a redirect. And then the browser sends all the data in the referrer header (if not deleted/changed by browser settings).

Ummm ..I can't see explaining what that means, working too well with Joe and Jane Sixspendallmydayonfacebookpack from geeimarriedmysisteraintshepurty, littleunderstandingville, somewhere,anycountry..
unfortunately :((

And incredibly they all get a vote too ( in countries where votes happen )..so politicians and legislators play to them..because they don't need bright voters either..bright ones might catch what is really going on ..

[Spiekerooger]

@Leosghost

Sad but true.

And as you pointed out as well about google using the white gloves of talking about ssl while starting to keep data only to themselves, we should name this thread "Google to stop giving referral data and moving to SSL" as the first is not a direct product by the second but this being rather two independent decisions by Google.

[wwconnect]

@Sgt_Kickaxe

I wonder though, Google knows that the FTC will eventually make a DNT (do not track) law mandatory where regular internet users must either opt in to being tracked OR have a readily accessible "don't track me" button that may even be on by default.

This is an excellent observation. Google may already know or be expecting a mandatory DNT law in the works or may be trying to preempt such a law by addressing the issue on its own. Whatever the reason blocking organic referrer data under the guise of security is a smart move on Google's part for several reasons:

1. Entice more people to search while signed in for "Security."
2. Increase awareness that it is even possible to sign in to Google.
3. With more people signed in Google gets more information on searchers .
4. More signed in people increases the likelihood searchers will use additional Google services such as Google plus.
5. With more people signed in Google can become a "curated" content portal for Internet users instead of just a search engine(i.e. you access customized iGoogle news feeds, your Google+ profile, your searches, your Google wallet, etc. the list goes on forever).
6. Disincentivize SEO for businesses by causing the value of AdWords to increase in relation to SEO.
7. Improve Google's organic ranking quality by reducing manipulation of results through SEO. SEOs will have the rug swept out from under them eventually.
8. Increase competitive advantage in the re targeting ad market. Actually, more like destroy the competition.
9. Create additional potential revenue for Google Analytics if they made the data available for paid users.
10. Steer the online privacy and tracking conversation by tacking control of the issue preemptively.

I could probably think of a bunch more unfortunately...

My favorite summary of what's going on right now and which I think is going to describe the next 6-12+ months is over on Aaron's book where he sums it up neatly saying, "Google Aggressively Enters Make Money Online Niche."

I really like Google and they have some great products. I'm also a big fan of Android. However, I make my living online and all I can say is that I really, really hope Google gets a nice fat *itch slap by the FTC here pretty soon. It would be nice that was followed by a nice swift kick in the *uts from the DOJ.

@walkman ... you should be on the Colbert Report.
but if there is anyone who didn't get that I think its safe to say you better start diversifying your online revenue channels beyond SEO/organic.

It would be cool to have some discussions on what the hot new areas are for making money online.

[J_RaD]

if goog is your primary traffic source... i'd say now is a good time to find another one.

[alvin123]

This deserves to be repeated. Thanks J_RaD

the jig is up....goog is just an enormous personal data sucking black hole that wants nothing more then the complete and total control of the net....and that means getting rid of ANYTHING they do not control, IE webmasters that run websites. Every single move they do is to erase something and replace it with "goog this"

[greenleaves]

That is why I treat G the way I do. Anything goes. Whatever works. They act that way; I do too.

To quote Daffy Duck;

This means war!

[kidder]

Anyway, it won't be difficult to throw an error message now saying "We're sorry, it appears that your logged into a Google account that is interfering with our website, please log out and refresh the page".

Nice!

[dstiles]

Comments on converting YOUR web site to SSL:

1. it slows down your delivery to both the punter AND to bots; so googlebot will take longer to process your site - will google worry? - and speed reports from browser tools will show a much slower page delivery. So all you guys who are trying to speed up your site to reduce googlebot's access time...

2. you will need a faster web server with higher bandwidth: there is quite a bit of extra data sent in SSL mode. And remember you will have to deliver your images in SSL as well because browsers are usually set to report mixed-source content, which will tend to turn punters away from your site.

3. it is feasible that not all bots are able to read SSL data anyway. Some of these bots may be amongst your stable of "permitted" bots.

[blend27]

I have a self coded Analytics program that makes a use of Search engine referrer data. Several years back I put a block in the code Due to this exact issue, Ref Contains GOOG Domain but "q=" as bank. The reason for this was that Scarpers were running bots from several Dynamic IP Ranges with that combination where "q" was blank. Hundreds of attempts of XSS, and SQL Injections.

I am aware of several high profile sites in my niche and plenty of others that have this rule in place. As a matter of fact the sites are ranking in top 5 for many many many terms.

The response to a blank q is a simple blank 403.

My personal opinion about this is that this is complete BS when it comes to a users Privacy. A company that wants my mobile phone number to Open an (G)email account.

I don't share data of my visitors on my sites(no GA, nothing), nor Google want to any more, I could understand, but nuking a base CGI Variable in a name of Profit? Common now…

this means war

Some one please comeup with a FireFox/IE extension that AutoMatgically Blocks all META redirects and JS Redirects, we will see how fast GOOG changes their minds.


That might break a few sites temporarily, but would BE under control a webmaster. GOOG is nothing without the content.

And I'll say it AGAIN, this has absolutely nothing with the user's PRIVACY, just GREED.

P.S: RUSTYBRICK ranks #1 for "site:twitter.com + google referrer data" on BING!

[universetoday]

We've done some tests and you still retain the keywords from an encrypted search when your website and tracker are also SSL.

Google Analytics can run through SSL, so it's theoretically possible to maintain this data.

You just need to purchase an SSL certificate, and switch your server over.

[Leosghost]

Spiekerooger found you couldn't ..the query was not passed when sent via SSL at google.com.

[EvilSaint]

For an industry that thrives on data to constantly improve and evolve, this is a big fat step back in time...by doing this, Google has just put webmasters back 5 years at least.

Incoming keyword data is what helps webmasters to improve their sites, pages and offerings.

Fair enough Google claims it only affects 9% of the visits. Still If someone gave you an apple which already had a bite taken off it, would you still eat it?

this means war

I think this statement represents how most of us are feeling right now towards Google.

Omniture, Coremetrics and the rest of the Analytics & Business Intelligence companies will be along with us shortly too...

The claim that it is for user protection is not validated especially if they continue to pass keyword information for paid ad interactions. So are Google saying that people using their adwords links aren't users who deserve privacy and protection?

We should start a campaign to turn off adwords advertising for a month! Make a pledge. Hurting Google's hip pocket is the one sure way to get them to notice and retract on this.

[Swanson]

Ok, I get people's point when they support the "privacy" issue - and that is Google's whole argument.

So why is privacy not important if you are running an Adwords campaign?

Very basically they are saying - "we pass that info if you are paying but not if you aren't".

Which means it fundamentally can't be about privacy. You can't have a strict privacy policy when you then allow the opposite if you are paying.

It is about SEO. It is about stopping SEO, just as Panda is.

Anyone who thinks it is for the user is ridiculously naive about the motives for this.

And the 1,000 keyword thing in Webmaster Tools - not useful for big sites that have millions of referring keywords.

Erm, what sort of sites will be affected?

[Swanson]

Just as a side note I was at the biggest affiliate conference in the UK this week where we got an insight from the biggest names in SEO and make no mistake there is a change in what they are talking about and it feels like there is a movement away from Google.

When you have these guys saying - ok, to get traffic from Google you need to be a brand, do all the things brands do and then there is no guarantee that it will work.

And then they say, they are re-positioning themselves to work outside of Google - i.e. social media, non-google traffic. One even said - they see a future without Google and how to look at working a business that doesn't need them.

You know there is now a massive disconnect and basically Google is no longer a viable source of traffic for these agencies who have a directive to enhance these brand sites.

There was little talk about Google in almost all sessions (including merchants and affiliates) - and open hostility from a certain major SEO in the one Google session with Pierre Far and Kaspar Szymanski, and a session after that in particular about the https masking of keyword referrals.

It's changed. And not just for affiliates - it's what these agencies are doing for major brands thats changed.

Google is in dodgy territory right now - don't get on the wrong side of the advocates that created their popularity.

The major agencies have the power to bypass Google and leverage facebook and other social media completely - and that includes Adwords. It only takes all of them to recommend campaigns totally outside of the Google infrastructure and they will become marginalised as fast as they have grown.

It's already starting to happen - there is only so much advertising budget and it is so seductive for brands to not only get new customers but at the same time create expanded communities on facebook to re-market to - Google just does not offer that. Agencies are now learning the platform and working out how to deliver immediate ROI and also create large facebook and twitter followings as well as increased email subscribers.

[Swanson]

I also want to add that one question from the panel to Pierre Far and Kaspar Szymanski about them noticing Adwords appearing at the bottom of the serps in a test was met with blank expressions and protestations about the fact they have no communication with the ad department and it's not their remit.

Now, I get the idea that they want to distance themselves as organic specialists away from the ad business.

However, as a business that operates in the real world does that not strike you as crazy?

I mean, if we were talking (as Google often does) about your website and the content of it and being in control of it - in what world is it okay to have part of your website controlled by different parties that have different motives?

Sites get banned from Google because of this.

You just can not have major real estate on the biggest search engine in the world controlled by two different un-communicating departments.

Does that not sound silly or am I totally missing something here?

They came across like little boys with toys and that was the common consensus from the guys I spoke to.

I appreciate the need to be impartial - but Google themselves talk about webmasters controlling their sites and the guidelines and the issues as a connected "site" thing.

If they don't connect (i.e. they don't communicate between departments that place content on their own massive site) then what the hell is that all about? Why do they expect us to?

There are 10 listings (content) on the search results page (sometimes all sorts because of universal search but you get the idea) and 8 sponsored listings.

That is a pretty even spread of content. And there is no communication between those two. So how do they do quality, conversion, user experience - simply how can they really test how good Google is for the user?

How can they really work with engagement metrics on Adwords if certain keywords could generate new universal search options on the main organic results - what happens if one day there is tonnes of local results with maps, and one day there isn't. What happens when different advertisers surface in Adwords that create higher clicks - what happens when advertisers use different creatives causing different click rates versus organic.

Without sharing between the two departments - how do you analyse cause and effect?

And if you don't communicate - isn't there a possiblility you get the wrong signals? Isn't it possible that Adwords advertisers affect the click rates of the organic - but that the dynamic quality of those advertisers then in effect change the click rates on the organic searches (i.e. the dynamic mix of brands etc. advertising on Adwords).

Surely Adwords and organic affect each other? So if you don't communicate between departments - how can you really create a Panda algorithm or any other one?

The answer is - either they are lying and they do communicate and correlate data.

Or they aren't and they don't. Now that is really very very scary and the algo is based on bad data.

Now my point is that no billion dollar business doesn't have half of their site not communicate with the other half.

So they can't be telling the truth. Someone must be looking at the bigger picture - someone is looking at interactions that include organic and adwords.

[edited by: Swanson at 2:53 am (utc) on Oct 20, 2011]

[Nuttakorn]

15% of searches we see everyday we’ve never seen before. No referral data, why would advertiser spend more.

[Leosghost]

G has always been pretty "siloed"..the only ones who seem to have access to all areas and know what is going on overall appear to be the lawyers , the accountants and the ruling triumvirate..

Follows the model of the medieval church very closely ..monks didn't talk to nuns except via the visiting bishop passing on what the archbishop wanted communicating ..and above the bishops who didn't have much contact with each other were the archbishops..again not much contact with each other ..above them the cardinals ..( a bit more contact between peers there ) ..and then the pope and inner confidents..top down dissemination of information and policy ..but with each member feeling that they were a part of a greater thing..a greater work..

When "same level" members met ..they didn't talk "shop" nor were they encouraged to do unless a superior was present ..and so on upwards through the system..

Word is.. inside Goog ..at the coffee machines etc .. different areas workers ..don't discuss what different areas are doing..part Chinese wall , part "deniability", part siloed management system ..but it has its roots a long long way back..

Perfectly possible that this referrer crap was not discussed even with some parts of the search team..the number of things they try out ..no way would they keep bringing everyone together and saying "hey guys ..what do you all think about we do this"..not enough hours in the 24 hour day for all the meetings that would need..

The number of times that their PR people who used to come here had to "go back" and "ask the engineers"..even Matt when he was GG here occasionally said "I'll have to go ask why that is"..

Inspite of his comments on this SSL and referrer thing ..he may actually not have been consulted at all about it ..he may have just been sent out to face the storm and to be "good 'ol Matt" .."smile and dissemble" ..'cos we'd have strung Eric up if he'd announced this one..

[Swanson]

Yep, I get it.

But monks didn't run a multi-billion dollar business.

Google is not a charity - everything it does now is commercial.

How does the click through component in adwords algo take into account the organic results for that keyword without knowing what organic results were presented.

Because that is the key - universal search was invented to change user behaviour - so as it is keyword dependent how does adwords know what affects click rates without knowing what was presented on the organic search?

If adwords doesn't know what search results are given for a query - how can it work out an optimal click rate or engagement rate?

And conversely if organic doesn't know what ads are presented for a query how does it know the affect of ads on engagement with its search results.

With the massive gulf in data how can you create a search algo that includes engagement?

You can't because other factors influence engagement.

You can only do it if you take the ads off the website and have it with a common purpose.