Keywords "not provided" trend and consequences

[aakk9999]

I have analysed a pattern of the "not provided" keywords on a couple of sites over the last 15 months. So far for March the first site has 32% of all organic visits arriving via "not provided" whilst the other has 37%.

The percentage has crept up from roughly 2% in Jan 2012 and the trend is up with increase averaging roughly 2% per month.

Here is graph for both sites:
[i46.tinypic.com ]

I am wondering if this will stop or will continue to increase. Having a third of organic traffic arriving via "not provided" causes serious problem in analysing any ranking issues. For example, should there be a drop in traffic, it is now impossible to analyse keywords and find out:

- whether the number of keywords driving traffic has dropped
- which keywords drive less traffic

as seeing decrease in either of these two metrics is greatly skewed by "not provided" bucket. A particular keyword "decrease" may not really be decrease at all as visitors arriving via this keyword may have their visit recorded in "not provided" bucket. Equally, drop in the number of keywords driving traffic is not saying much either, as all these "lost" keywords may actually be present in "not provided" bucket.

Has anybody else looked at percentage of visitors that arrive to the site via "not provided" keywords? Are your numbers similar?

I have started to download and save the query data from WMT to have some historical information as WMT (as well as Search Engine Optimisation -> Queries in GA) only show the last three months of keywords.

Anyone has any other idea on how to "recover" the data we used to have?

[Robert Charlton]

Have you been looking at how much of your traffic is mobile traffic, or coming from wireless devices like tablets? Filtering by user agent should help you get a handle on this. I would assume that as mobile increases, and it will, "not provided" will increase as well.

This thread from Oct 2012 touches on how fast mobile traffic has grown....

43 Percent Of Total Google Search Queries Are Local
http://www.webmasterworld.com/local_search/4504560.htm [webmasterworld.com]

In my post on the thread, I cite several sources (including this forum's Sept 2012 Update thread), that have suggested how fast mobile traffic is growing.

[lucy24]

But, but, but...

Concealing search terms is not an inherent and unavoidable property of mobile devices. It's a choice made by the search engine.

Question: Do they understand that you need to know those search terms in order to make your site better? It isn't just about ranking for good words. It's about not ranking for bad ones. The difference between real sites and Pure Spam sites-- yes, OK, one difference --is that if your site is about cutting-edge techniques of uranium extraction, you don't want to come in at #1 with people seeking information on the dietary habits of polar bears.

:: obligatory detour here to make sure nothing in the above paragraph would bring me bobbing to the top-ten surface ::

And if you are getting those obviously-wrong hits, you don't want the search engine fiddling with your page title to make it seem as if it's just what the searcher wanted. You want to know what words are involved so you can adjust your page accordingly. ("Oh, oops, didn't realize that my favorite widget synonym is also a wideley used technical term among the foobar crowd. Better use a different word.")

This, in turn, will make the search engine look better. Um. Won't it?

[phranque]

the "choice made by (google)" was to "not provide" keyword search terms for visitors using secure protocol to search. (for example searching while "logged in to google")
the number of searchers using secure protocol has increased steadily since google implemented this policy in october, 2011.

i believe the reference to mobile search implies that these visitors are typically logged in while searching from mobile devices.

[aakk9999]

I have checked and only a 25% of "not provided" arrive via mobile, which makes it around 8% of all organic search. I have noticed though that once logged out, Google stays in https for future searches.

But this does not answer the question on what is the best solution to overcome this lack of keyword information in site analysis - other to keep exporting WMT queries regularly.

[netmeg]

I suspect that eventually ALL organic search terms will be "not provided". Yea, I said it.

[deadsea]

My understanding of this was that it is actually implemented by browsers. Browsers don't send full referrer info when clicking from a http to an https site. Browsers only send the domain and protocol, but not the path and query string in the referrer.

I believe you can get full referrer info from Google by fully hosting your site on https.

[netmeg]

I believe you can get full referrer info from Google by fully hosting your site on https.

Not as far as I know. Got a reference for that?

[TheOptimizationIdiot]

My understanding of this was that it is actually implemented by browsers. Browsers don't send full referrer info when clicking from a http to an https site.

Exactly, except I think you meant from https to http.

Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol.

[w3.org...]

[lucy24]

Oh, that reminds me of what I meant to ask. In the original post, were you talking about

--searches from https google, where there is no query string at all in your referer

or

--searches that send a complete query string including cd= (i.e. where your result ranked), omitting only the search term itself

or both?

[aakk9999]

^^^ I also use statcounter and I can see that in most cases there is a referrer string, only the q= parameter is empty.

[Shepherd]

I believe you can get full referrer info from Google by fully hosting your site on https.


Not as far as I know. Got a reference for that?

That is interesting, I just did a quick look at some stats for ENTRY pages on one of our sites that are secured that received organic traffic from google directly to the secured page and I could not find any "not provided" keywords, every one I looked at showed the keyword search that led to the secured page.

Wonder if switching the whole site to https might eliminate the "not provided"... mind racing now...

[TheOptimizationIdiot]

Wonder if switching the whole site to https might eliminate the "not provided"... mind racing now...

Mine too. Definitely sounds plausible and I've recommended people "go secure" as long as 18 months ago, but now it seems there are even more reasons for it.

The toughest thing I'm having to figure out is how to minimize rankings impact while 301ing everything from http to https. I really think it should have been done on the site I have in mind years ago, now I'm not sure of what the best answer is, but I'm almost positive it's going to have to happen at some point in time.

[lucy24]

I just did a quick look at some stats for ENTRY pages on one of our sites that are secured that received organic traffic from google directly to the secured page and I could not find any "not provided" keywords

Look directly at your logs. All you need is one https google referer. Maybe two or three to make sure. If it has a query string, then you are right.

Which in turn means we are blaming google for something that isn't really google's fault-- apart from using https in the first place, which seems like overkill. To me, https means the site is going to ask for my credit-card number.

What about those empty "q=" slots? If you get https google to cough up a query string, does it include the full q=something-here element?

[Shepherd]

All you need is one https google referer.

couldn't find one in the last 1000 visits (all my eyes could handle right now), found a couple of [google.com...] referers but nothing with any data.

[Leosghost]

IIRC ( but I cant find the place to quote easily due to the complete rebuild of this machine, the HDs used at the time are currently not in any machine, and my "HD dock is in need of repair ) Google ( at the time of implementing "logged in" searches done via https )said that adwords advertisers who hosted on https would get the KW passed to them when searchers used https..but that merely hosting on https would not provide it..

I tend to agree with Netmeg on this ..I too think they'll phase out all passing of query string data to non adwords customers..Why would they continue to pass us data that we can use to help us reach page one above the fold..when they can sell us a spot there ..and make achieving a page one spot even more hard by non paying methods, by making it harder to tell how one gets there, when one does..

re mobiles ..as most ( but not all ) android users stay logged into their android ( Google )accounts when using their android phones probably just because they forget they are "in" )..this is why they are showing as https searches..iphone users and non android searches will not normally exhibit this behavior..

[Robert Charlton]

i believe the reference to mobile search implies that these visitors are typically logged in while searching from mobile devices.

That was my assumption, which was correct as far as it went. The logged-in/ SSL issue isn't new, but my interpretation of its behavior was simplistic. There are some "paradoxes" (which is to say "inconsistencies") involved in Google's SSL implemention, plus some other twists, which have made any simple interpretation difficult.

Until recently, it turns out, Google did not automatically shift mobile users to a secure connection on Google.com, since latency issues made an SSL connection too slow. But, as observed by Ryan Jones and researched further and reported by Danny Sullivan (references below)... at the end of Sept 2012, Apple upset the cart and surprised Google by stripping all referrer data for searches made from the Safari search box on iOS6. This made searches from the Safari search box (on iOS6 only) look like direct traffic rather than search traffic. The shift began around 9/19/2012.

Read...

Google not sending any referer data on iOS default search?
September 26, 2012 - by Ryan Jones
[dotcult.com...]

and...
Safari Shifts To Google Secure Search in iOS 6, Causing Search Referrer Data To Disappear
Sep 27, 2012, by Danny Sullivan
[searchengineland.com...]

In this and the follow-up articles, Danny also comments on question of https to https referrer data. As I remember, he made the point that Google isn't using straight https, so the referrer data, which might otherwise be retrieved by an https site at the receiving end of a two-way https connection, is in fact intercepted by Google. I've seen it mentioned, but haven't read in depth, that Android 4.0 may also be dropping referrer information.

On desktop, some browsers are defaulting to https, and Google is keeping users of its various services signed in and there's increasing referrer data that's lost.

[indyank]

Goal - 100% no referral data for non-adword advertisers.

Though google doesn't intercept and wipe out https to https referral data for adword customers, like they do for others, none can get 100% referral data.

[sergeivladimov]

Hello,

I heard that there's something to do with HTTP/HTTPS or logged-in/logged-out in Google Search the triggers the keyword hiding.

But I tried with Safari and Firefox, HTTP or HTTPS, logged-in or not, I can't get the keyword passed. For any result I click, Google first directs me to an intermediary page like:

http://www.google.com/url?sa=t&rct=j&q=&source=web&cd=4
&ved=[hidden]&url=http%3A%2F%2Fexample.com%2F&ei=[hidden]&usg=[hidden]&bvm=[hidden]

As you can see, q= has no value. The source code of this page is simple:

<script>
window.googleJavaScriptRedirect=1
</script>

<script>
var m = {
 navigateTo: function(b,a,d) {
 if (b!=a&&b.google) {
 if (b.google.r) {
 b.google.r=0;
 b.location.href=d;
 a.location.replace("about:blank");
 }}
 else {
 a.location.replace(d);
 }}};

m.navigateTo(
 window.parent,
 window,
 "http://www.example.com/");
</script>

<noscript>
<META
 http-equiv="refresh"
 content="0;URL='http://www.example.com/'">
</noscript>

So what it does is basically overwriting the referrer set by the browser by redirecting me with a script or a meta on an interstitial page but not a HTTP redirect. And this interstitial page does not have the keyword in its URL.

I see this everywhere on Google Search results. So is this it, did they just went 100% "keyword not provided"?

Thank you.
.

[edited by: Robert_Charlton at 10:41 pm (utc) on Sep 23, 2013]
[edit reason] fixed sidescroll [/edit]

[lucy24]

So is this it, did they just went 100% "keyword not provided"?

:: quick panicked detour to g### ::

Here's what LiveHeaders shows. I am NOT logged in and FF is not my default browser, so this is what you now get if you simply type in www dot google dot com. In summary:

https://www.google.com/

GET / HTTP/1.1
Host: www.google.com

HTTP/1.1 200 OK

[google.com...]
sfa,shb,tbpr,hsm,pcc,csi/rt=j/ver=Foo5jL-bxec.en_US./am=AAI/d=1/sv=1/rs=AItRSTObnbIho8ud38zdeCxTRyRJwHuWiA

GET /xjs/_/js/s/sb,cr,cdos,tbui,mb,wobnm,cfm,abd,klc,kat,bihu,kp,lu,m,rtis,tnv,amcl,hv,lc,ob,r,rsn,sf,sfa,shb,tbpr,hsm,pcc,csi/rt=j/ver=Foo5jL-bxec.en_US./am=AAI/d=1/sv=1/rs=AItRSTObnbIho8ud38zdeCxTRyRJwHuWiA HTTP/1.1
Host: www.google.com

Don't look at me. That's the browser, not me.

HTTP/1.1 200 OK

[google.com...]

GET /xjs/_/js/s/sy9,gf,sy145,vs/rt=j/ver=Foo5jL-bxec.en_US./am=AAI/d=0/sv=1/rs=AItRSTObnbIho8ud38zdeCxTRyRJwHuWiA HTTP/1.1
Host: www.google.com

HTTP/1.1 200 OK

[clients1.google.com...]

GET /generate_204 HTTP/1.1
Host: clients1.google.com

HTTP/1.1 204 No Content

wrf?

https://www.google.com/csi?v=3&s=webhp&action=&e=17259,39523,4000116, {buncha stuff snipped here}

GET /csi?v=3&s=webhp&action=&e=17259,39523,4000116, {et cetera as above}

HTTP/1.1 204 No Content

[clients1.google.com...]

GET /complete/search?client=hp&hl=en&gs_rn=8&gs_ri=hp&cp=1&gs_id=8&q=i&gs_gbg=o065383l7REB9i616ux HTTP/1.1
Host: clients1.google.com

HTTP/1.1 200 OK

[clients1.google.com...]
&gs_id=n&q=in&gs_gbg=O072fRNPcefUpSUd2FaHg2FMHuBGB4C

GET /complete/search?client=hp&hl=en&gs_rn=8&gs_ri=hp&cp=2
&gs_id=n&q=in&gs_gbg=O072fRNPcefUpSUd2FaHg2FMHuBGB4C HTTP/1.1
Host: clients1.google.com

HTTP/1.1 200 OK

.... anyway, it goes on like that for a total of NINETEEN iterations.

After that I finally get my actual search, followed by several more rounds of

https://www.google.com/xjs/_/js/s/sy9,gf,wta,sy48,sy49,sy50,sy52,sy56,sy51,sy57,sy53 et cetera

And that's all one search.

The sequence includes several of these:

https://clients1.google.com/generate_204

GET /generate_204 HTTP/1.1
Host: clients1.google.com

"I'd like a 204 please." "Sure, here you go."

Finally I get to the request for my own page, which is what I was testing for:

http:/ /www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDwQFjAA&url={my page URL here}&ei=ZGZiUa4UioWIAtvLgMgK&usg=AFQjCNF2eEf6FMrTbAGRL4EE3gdljTOYPg

So, yup, that's a "q=nothing-at-all".

q=If they're not going to tell you what the question was, why bother to include the 'q=' bit in the query string at all?
.

[edited by: Robert_Charlton at 6:39 am (utc) on Sep 24, 2013]
[edit reason] fixed sidescroll [/edit]

[deadsea]

I also just did some testing with live HTTP headers. I searched for something that was likely to have HTTPS only sites in the search results, and I clicked on one with live HTTP headers recording everything.

When logged in, even when I click on a HTTPS site, the referrer that Google sends does not contain the keywords. Like Lucy, I see a blank q= parameter sent to the HTTPS site.

[TheOptimizationIdiot]

I've still got a few keyword queries from Google today, it'll be interesting if those "phase out" over the next little while.

I'm sure if they're really going to (get to) not providing any keywords we'll get a nice spin about how it's not to keep us from knowing or them from getting gamed, but rather to protect the privacy of their searchers, because we all know from their track record with Buzz and that little Street View Wifi Resording issue and a number of other "little oopses" they're all about personal privacy, er, uh...

[lucy24]

I detoured here to confirm a hunch originating from another thread: Since g### uses UA detection rather than feature detection, the hidden-search-term referer should come up in all visits from "new" browsers. Which in turn would mean that people whose clientele includes a lot of old-browser users (say, rich conservative elderly people who buy antiques or real estate ;)) are sitting pretty.

I was wrong. I fed a search string into http google dot com, not logged in. Only firefox gives the &q= {nothing} version. Safari, Opera and Chrome (!) show me my query loud and clear.

Interestingly MSIE doesn't. This is 5.23 so perhaps they have a separate Search routine for beyond-ancient browsers; in this version you get "q={name-of-page}". Camino gets Search Option B because g### thinks it's an older browser; the query string in the referer includes "&gs_l=heirloom-hp.3 {and a bunch more stuff}". No information from Lynx: turns out it doesn't send a referer at all.

Does anyone know what "ved" is? Interesting pattern; three of my six browsers gave the identical content and two others were similar. (MSIE included the query but Camino didn't. Odd.)

[Dymero]

Still getting keywords today, though now ~30% of my organic traffic is (not provided).

[tedster]

~30% of my organic traffic is (not provided)

That's what I see on many sites, too.

[Robert Charlton]

For more discussion on keywords data, see this progression of threads... two on the keywords tool and more difficult access to the keywords popularity data that Google was providing...

Google: Keywords tool will not be available without Adwords account
July 3, 2013
http://www.webmasterworld.com/google/4589828.htm [webmasterworld.com]

Google Keywords Tool External Now Gone
Aug 27, 2013
http://www.webmasterworld.com/google/4605230.htm [webmasterworld.com]


...and this discussion about Google shutting down the organic referral data....

Not Provided - Looks like organic keyword data is gone
Sept 23, 2013
http://www.webmasterworld.com/google/4612072.htm [webmasterworld.com]