homepage Welcome to WebmasterWorld Guest from 54.197.111.87
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Changes to Google referrer starting this week
pavlovapete




msg:3892575
 1:22 am on Apr 15, 2009 (gmt 0)

... we do want to let you know about some changes to Google search that are coming down the pike, before you start seeing (potentially) alarming headlines.

[analytics.blogspot.com ]

New format:

http://www.google.com/url?sa=t
&source=web
&ct=res
&cd=7
&url=http%3A%2F%2Fwww.example.com%2Fmypage.htm
&ei=0SjdSa-1N5O8M_qW8dQN&rct=j
&q=flowers
&usg=AFQjCNHJXSUh7Vw7oubPaO3tZOzz-F-u_w
&sig2=X8uCFh6IoPtnwmvGMULQfw

I'd be interested in your opinions about meaning of the new format. I gather these random strings are some kind of (encrypted) signature to combat something undesirable or ensure some desirable outcome?

Cheers

[edited by: tedster at 1:25 am (utc) on April 15, 2009]
[edit reason] de-link the sample format [/edit]

 

tedster




msg:3892577
 1:33 am on Apr 15, 2009 (gmt 0)

As the article explains:

The key difference between these two urls is that instead of "/search?" the URL contains a "/url?". If you run your own analyses, be sure that you do not depend on the "/search?" portion of the URL to determine if a visit started with an organic search click.

Google Analytics does not depend on the "/search?" string in the referrer, so users of Google Analytics will not notice a difference in their reports, but other analytics packages may need to adapt to this change in our referrer string to maintain accurate reports.

Also, they are rolling this out initially on a small scale - mindful of the recent analytics problems from the Ajax search results [webmasterworld.com].

Now the "why" for these changes comes into play, right? This blog post only comes from the Google Analytics team and their focus is not on the "why", just the "what".

My bet - this is the next step toward whatever the original Ajax results were trying to accomplish, including future enhancements to the results page itself, adding various kinds of onmouseover or onclick information and still preserving the Analytics information.

Robert Charlton




msg:3892689
 6:29 am on Apr 15, 2009 (gmt 0)

I wonder if the old search urls will redirect to the new ones. I'm thinking of my own bookmarks, as one example, and of many links on the web, in existing documents, etc, that reference searches.

mirrornl




msg:3893271
 9:31 pm on Apr 15, 2009 (gmt 0)

Matt Cutts confirms that Google is adding ranking data to referrer string.
I suppose this link is allowed:
[analytics.blogspot.com...]

<snip>

Could be bad for affiliates who drive most of their traffic from organic search on Google. If the operators know what the keyword was from analytics and the placement from the referrer string why won't they just optimize for the terms themselves?

[edited by: Receptional_Andy at 10:35 pm (utc) on April 15, 2009]
[edit reason] Moved from another location + no specific keywords, please, see forum charter [/edit]

Receptional Andy




msg:3893273
 9:42 pm on Apr 15, 2009 (gmt 0)

This is interesting. The example URL given in the blog post above is www.google.com/url?sa=t&source=web&ct=res&cd=7&url=http%3A%2F%2Fwww.example.com%2Fmypage.htm&ei=0SjdSa-1N5O8M_qW8dQN&rct=j&q=flowers&usg=AFQjCNHJXSUh7Vw7oubPaO3tZOzz-F-u_w&sig2=X8uCFh6IoPtnwmvGMULQfw

Now, Google already send clicks on search results through those URLs - via a redirect. this is either triggered via javascript when you click on the link, or is occasionally hard-coded when, I assume, they're testing how the javascript method is working out.

But, these URLs never show up in referrers, since they go through a server-side redirect. For it to be a referrer, it would presumably have to be a browser-side redirect like a meta refresh or some kind of javascript.

So, how are these URLs supposed to suddenly start ending up in referral strings? How will a browser end up on a URL like the above after searching for something?

centime




msg:3893278
 9:49 pm on Apr 15, 2009 (gmt 0)

surely the information stops at the site of the affiliate.

The referrer information passed to the merchant would still be affiliates web address and server details.

Then again , maybe i misunderstand analytics

jdMorgan




msg:3893308
 10:15 pm on Apr 15, 2009 (gmt 0)

I've occasionally seen this new google.com/url referrer format in my server's raw access log file since 22:50 GMT on April 10th. So it seems that the blog post came four days after the initial roll-out.

However, it looks like some of the query parameter names have changed, so I may have just been seeing a 'beta test.'

Jim

Receptional Andy




msg:3893316
 10:24 pm on Apr 15, 2009 (gmt 0)

If you visit the example URL, you'll see a "redirect warning" asking whether you want to go to the target URL. Of course, such a page would create a new referrer. However, the only reason you see the interim page is because the destination doesn't match the checksum in the URL (Google added this test, incidentally, because phishers and the like were hijacking their redirect system to send people to convincing "google.com" urls).

There are only so many reasons I can think of why site owners would see the new referrer as essentially a redirect URL, like a bug in the click-tracking script, some kind of wacky interface changes on Google serps, or it's all a false alert ;)

g1smd




msg:3893336
 10:42 pm on Apr 15, 2009 (gmt 0)

The whole idea of a referrer is that it is the URL of the page the visitor came from.

This URL is NOT the URL of the search page the user was on. What a mess.

dstiles




msg:3893343
 10:55 pm on Apr 15, 2009 (gmt 0)

There is no reason why a referer cannot be altered by the redirecting server. Any or all of the header parameters can be changed.

In theory there is no reason why clicking on a google result link couldn't result in a referer saying it's from yahoo. :)

Receptional Andy




msg:3893360
 11:23 pm on Apr 15, 2009 (gmt 0)

Not quite, dstiles - the referrer is supplied by the browser, not the server. A client (i.e. web browser) could spoof a referrer from a different domain, but a server couldn't. The browser will need to physically load Google's new URL and pass it on to the destination site in the referrer header.

dstiles




msg:3894104
 9:45 pm on Apr 16, 2009 (gmt 0)

Not if the request is redirected through a server somewhere. Which google does.

I use the techique here for testing my sites with various header parameters without continually altering the browser.

Receptional Andy




msg:3894122
 10:01 pm on Apr 16, 2009 (gmt 0)

Not if the request is redirected through a server somewhere. Which google does.

I use the techique here for testing my sites with various header parameters without continually altering the browser.

Servers supply responses to client requests. It is the request headers (supplied by the client/web browser) that contain a referrer. Web servers have no control whatsoever over this value. Web browsers determine what to report - usually the last "page" URL visited.

Servers can send browser to alternate URLs, but that means the browser must visit that URL. So Google can't send people just anywhere to influence referrers - they have to send the browser to a valid URL for it to ever show up in a referrer.

It's a shame there's no clearer definition of what should be in a referrer other than this:

The Referer[sic] request-header field allows the client to specify, for the server's benefit, the address (URI) of the resource from which the Request-URI was obtained (the "referrer", although the header field is misspelled.) The Referer request-header allows a server to generate lists of back-links to resources for interest, logging, optimized caching, etc. It also allows obsolete or mistyped links to be traced for maintenance. The Referer field MUST NOT be sent if the Request-URI was obtained from a source that does not have its own URI, such as input from the user keyboard.

HTTP/1.1: Header Field Definitions [w3.org]

And for the new Google referrers to work, it would require the /url? URLs to eventually send users to the search result they clicked on.

I still can't imagine what must happen on result pages for this URL to be the eventual referrer. Some kind of wacky javascript is about my best answer. But then, I see quite a few Google referrals that omit the 'q' parameter these days (I always thought that was the key one!) so who knows what they're up to?

dstiles




msg:3894145
 10:39 pm on Apr 16, 2009 (gmt 0)

If the target is a server (eg google's redirect server) then it's possible to alter header fields as the request travels through it. It's just another form of proxy.

So-called "privacy services" are constantly removing or modifying headers before passing on the information to the server and ditto on the way back to the browser.

I submit a request to my proxy saying I want a certain page from my web site; the proxy server modifies the headers to what IT wants; the proxy sends the modified request to the target server/web site; the target web site behaves according to what the proxy tells it, not what the browser requested, and returns the content - or not; the logs reflect accordingly.

Works here, anyway. :)

Obviously in google's case most, if not all, of the original referer (or whatever) string gets passed on to the web site, but it doesn't have to be. And again obviously, this only works if the original request goes through an intermediate server of some kind, be it proxy, firewall, privacy service or whatever.

Receptional Andy




msg:3894167
 11:14 pm on Apr 16, 2009 (gmt 0)

Again, not quite ;)

A proxy is just a client between your web browser and the server. And like any client, it can supply whatever data it likes to the server. This would only work with Google results if, when your browser requested www.google.com/url?something it sent another request to the actual search result destination. But that would be the equivalent of scraping the page of sites in search results, and browsers never leaving Google. Would Google even consider such an extreme measure?

Redirects are different - they send the client to a different URL.

Don't forget, what we're talking about here is the referrer as sent to sites that are clicked in search results. These are specified entirely by end-user's web browsers when they arrive at the search result destination. These browsers must visit the URL www.google.com/?url=something for that to appear in a referrer request header (unless end-users suddenly configure their browsers to spoof referrers. That URL can deliver whatever content it likes, though, of course.

Receptional Andy




msg:3894534
 1:02 pm on Apr 17, 2009 (gmt 0)

In any case, I've seen an example now. Google are using the following code on the /url pages:

<script>if(parent!=window&&parent.google){parent.location.href='http://www.example.com/example-url';location.replace('about:blank')}else{location.replace('http://www.example.com/example-url')}</script><noscript><META http-equiv="refresh" content="0;URL=http://www.example.com/example-url"></noscript>

From: www.google.com/url?sa=U&start=2&url=http://www.example.com/example-url&rct=j&ei=[checksum]&q=example+keyword&usg=[checksum]

So basically the click on a result leads to a page with a javascript redirect or a meta refresh if scripts are disabled. This particular example did not include any useful additional data (e.g. rankings) which has been reported as available in some cases.

One annoying side effect is that you can no longer go back to results to see how the listing appeared. You would have to guess at the particular options used in the search, etc.

[edited by: Receptional_Andy at 1:03 pm (utc) on April 17, 2009]

dstiles




msg:3894966
 10:52 pm on Apr 17, 2009 (gmt 0)

A proxy is not necessarily a client; public stealth proxies are servers and DO alter headers.

A redirect can be as simple as (eg) google taking you to their server, modifying the headers and then redirecting (with the new headers) to the target site. There would be no point in going to the target site first.

I'm not saying google does that, only that it's possible IF they redirect through their servers to the target (how does PPC register clicks? I assume this way). It's exactly what a proxy does, to one degree or another.

The results I'm seeing listed on google here are in the "old" format, not redirected, so I can't comment on what google actually does do with a redirect. I'm just saying that if they DO redirect via one of their servers then changing the referer etc is possible.

... Just read your last posting. That is not a redirect as I assumed for this discussion. I was assuming that by redirect you meant through an intermediate server.

I have to say that wouldn't work for all browsers. Increasingly people are turning off javascript because of dangerous exploits. The only reason I allow it on google is for auto-numbering of results (which is currently broken so...) and most other sites I only turn it on if I want a specific function and I REALLY trust the site.

The meta refresh WOULD work IF something like NoScript was not blocking redirects.

To be honest, I'm not entirely sure where your code sample is located. I assume it has to be on an intermediate page between the results page (where you click) and the target site? So what does the actual google result link look like?

I suppose google will roll out this in my neck of the woods at some point. :)

Receptional Andy




msg:3894981
 11:16 pm on Apr 17, 2009 (gmt 0)


A proxy is not necessarily a client; public stealth proxies are servers and DO alter headers

I don't think we're going to agree about proxies ;)

As far as Google results goes, this is the way it's worked for a while:

- Google output the HTML of the results page
- In the HTML, links to results are plain old hrefs straight to the page
- A javascript function captures onclick events, and when you click on a result the script changes the destination to their own tracking URL
- Occasionally, Google hard-codes the click-tracking links in SERPs to go directly to the tracking URL (I assume to compare data with the javascript method, to make sure it's working adequately)

Google.com/url is the location of their click-tracking script. It grabs parameters in the URL for various purposes - mostly for tracking (the ranking and a few other bits and pieces - whether you clicked "did you mean?", which results page, etc).

There are also checksums that try to validate that they created the tracking URL, not you (otherwise you can hijack their redirect script, which was happening to them for a while). If the checksum doesn't match, you see a "we're sorry" page. It breaks if certain parameters aren't present, since it hasn't been programmed with an accurate response to that:

[google.com...]

If it's a valid URL, the script uses a location: response header, triggering a 302 HTTP redirect (with the effect of sending the browser to the actual result URL). The change is using javascript/meta refresh to redirect, instead of outputting a HTTP redirect. I think many would agree with me that this is generally seen as a unsatisfactory method of achieving a redirect.

Browsers don't typically update the referrer they send if they follow an HTTP redirect - they will usually use the prior URL as the referrer (perhaps because a redirect URL is not a good place to point people to). But javascript/meta redirects trigger a page load, and browsers may use that as the referrer they send instead.

But I'm inclined to think the referrer is just a side effect of the change, which is for some other reason. The click tracking script is nothing new (see [google.com...] for just a few of the posts about it). But tracking URLs showing up as the referrer certainly is.

dstiles




msg:3895311
 8:16 pm on Apr 18, 2009 (gmt 0)

I admit proxies was a bit of a red herring here. I mis-understood the method of redirect. Sorry about that, Andy. :)

I understand now what you're saying about the javascript redirect. I also note your comment re: hard-coded redirect for checking, presumably to a kind of proxy? Sorry. :)

So, with JS turned off their tracker fails and the direct url takes over. That makes sense. I know browsers don't usually change the referer on a redirect but there is no reason why not other than convention, I suppose, as the point of redirect may be virtual.

Does anyone know if the referer changes AT ALL between JS enabled and disabled?

One thing is sure: google MUST retain a system that does not rely on JS OR meta redirect to work.

The brewing storm over Phorm, spreading now from UK to world-wide, is not only going to bring attention to communications interception but, thanks to Phorm's counter-argument, to google's method of obtaining, retaining and using tracking information. I suspect more people in the near future will become cautious about JS in particular and redirects by implication, if for no other reason than because NoScript can easily turn them off. An increasing number of people are using NoScript (46,104,097 downloads to date and rising at the rate of 600,000 a week) and with the current publicity over privacy I suspect this will rise rapidly in the UK if nowhere else.

g1smd




msg:3895378
 12:55 am on Apr 19, 2009 (gmt 0)

*** One annoying side effect is that you can no longer go back to results to see how the listing appeared. You would have to guess at the particular options used in the search, etc. ***

Yes. That's the exact point I was making above. :)

oddsod




msg:3902585
 6:17 pm on Apr 28, 2009 (gmt 0)

What nobody seems to mention is why some links in the SERPs are picked out for this treatments and others not. Is it random or can we read anything into this (and we have a limited window before all SERPs links move over to the new format)?

I did notice that when I'm logged into a Google account I'm more likely to see the new URL for my own sites. Also, when you have sitelinks, the sitelinks have the new URL but the link to the homepage doesn't.

tedster




msg:3902626
 7:17 pm on Apr 28, 2009 (gmt 0)

As far as I know, sitleinks have always included a tracking redirect for the link.

This new kind of link is just being rolled out and tested - so I don't think there will be a solid way to know when it shows up and when it doesn't, at least not for a while.

tedster




msg:3908412
 4:13 am on May 7, 2009 (gmt 0)

Note this tidbit - the ranking information in the new style of referer includes "universal search" items. So if there are three Image Search results, followed by your result (really in the first organic position) the parameter will report the click as happening on position #4.

reference: Google Help Video [youtube.com]

Robert Charlton




msg:3908465
 7:20 am on May 7, 2009 (gmt 0)

tedster - Thanks for that video link. In addition to explaining how Google will be reporting positioning, Matt Cutts also mentions in the video (at about 50-seconds in) that the new search string will have the query embedded in it.

This partially addresses the question I had above, which was prompted by the lack of the query in the search string as it was originally announced.

It's a must-see video, 3-minutes long.

enigma1




msg:3952345
 4:37 pm on Jul 14, 2009 (gmt 0)

After reading the comments here and doing a bit of testing it is possible to scrap content from sites and the site owner looking the server logs will believe that everything is a legitimate request.

So first of all, the referrer link from google is always valid regardless of geo location. If I take a referrer link I see in my logs and place it on my browser I get a redirect either via jscript or meta-refresh (if js is off).

Now a server could simply setup that link of the page he wants to scrap and use one of his clients to do the job. So the client can act as the man in the middle, assuming scripting is enabled on his browser, contact the target server retrieve the content and pass it over to the scrapper via js without even knowing it.

At the moment to get around it, I deployed a form, if the web application detects a redirection script on the referrer field. Then a visitor on my site needs to click a button to continue to the real page. This seem to work, but of course it may cause other side-effects I haven't yet considered.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved