Lost referrer

Loosing HTTP referrer in FireFox when user is coming from MSN

[cymbal]

We are having a very odd issue. When a user is using a Mozilla browser and visits our site from MSN or from the Yahoo Site Explorer, the HTTP Referrer is empty. IE and Opera work fine but FireFox, Flock and Netscape don't.

We have tested this on multiple machines. The same machines visiting the site from Google or Yahoo search have the referrer.

Anyone else seeing this?

[encyclo]

That's an interesting observation - it appears to be to do with onclick events which remove the referer details during the subsequent redirect. Looking at MSN Search specifically, each site link is overridden by a Javascript function calling "GLinkPing.aspx":

<a href="http://www.example.com/" gping="/GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&amp;&amp;DI=293&amp;
IG=aeda64ef9eb548638565d31309e0e1f1&amp;POS=1&amp;CM=WPU&amp;CE=6&amp;
CS=AWP&amp;SR=6&amp;sample=0">example</a></h3>

(Carriage returns added for clarity, some details obfuscated)

If you enable referers in the browser and click the link, you can record the HTTP headers. You see something like this:

http://search.msn.com/GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&&DI=293
&IG=aeda64be9eb548638565d31309e0e1f1&POS=1&CM=WPU
&CE=6&CS=AWP&SR=6&sample=0

GET /GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&&DI=293
&IG=aeda64be9eb548638565d31309e0e1f1&POS=1
&CM=WPU&CE=6&CS=AWP&SR=6&sample=0 HTTP/1.1
Host: search.msn.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-ca,en;q=0.8,fr-ca;q=0.5,fr;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
[b]Referer: ht[/b][b]tp://search.msn.com/results.aspx?q=example&FORM=QBHP[/b]
Cookie: [i](removed)[/i]

[b]HTTP/1.x 200 OK[/b]
Content-Length: 42
[b]Content-Type: image/gif[/b]
X-Powered-By: ASP.NET
P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND", policyref="http://privacy.msn.com/w3c/p3p.xml"
X-TraceID: b1f123f02fac4849968f6212fef11b81
Expires: Sat, 09 Sep 2006 17:52:06 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sat, 09 Sep 2006 17:52:06 GMT
Connection: keep-alive

----------------------------------------------------------

http://www.example.com/

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-ca,en;q=0.8,fr-ca;q=0.5,fr;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

HTTP/1.x 200 OK
Date: Sat, 09 Sep 2006 17:52:07 GMT
Server: Apache
Last-Modified: Sun, 03 Sep 2006 21:17:51 GMT
Etag: "2860310-bca-44fb467f"
Accept-Ranges: bytes
Content-Length: 3018
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

At the moment, I'm not entirely sure what the GLinkPing.aspx script is doing, but it declaring a MIME type of

image/gif

- it is actually a 1x1 pixel transparent GIF image rather than a HTML document. A tracking beacon, maybe, which remains in the browser cache? As you can see, the

gping

attribute is working as an onclick event, overriding the link. Once the image script is fetched, there is a redirect to the chosen website - but in Firefox (and thus in other Gecko-based browsers), the referer information is not included in the second request.

As there is no 301 or 302 redirect (only a 200 OK for GLinkPing.aspx), perhaps Gecko doesn't realise that it should pass on the information to the landing page? Perhaps the image MIME type is confusing things?

Does anyone get any referers from MSN Search from Gecko browsers (Firefox, etc.)? If referers are never passed, then we may be underestimating the number of visitors coming from MSN - even if you consider the probability that most MSN Search users are connecting with Internet Explorer.

[tedster]

I just ran filters over a couple hundred logs from June, to present -- and it seems to be true. I cannot find a Gecko browser with a referer of search.msn.com

[carguy84]

leave it to Microsoft to add in their own, custom, HREF attribute...

[kaz]

well. this is true. good to know at least.

[bmcgee]

I haven't checked lately but as of a few weeks ago my own clicks to my sites from msn and using firefox were just fine. I won't say that I know it is in the logs but my asp tracking picked up the referrer just fine.

I don't see a problem...

[kaz]

You should check now bmcgee, i am not sure yet when ... but currently ...

[surfin2u]

Opera has an option to enable/disable referrer logging. I wouldn't be surprised if FireFox does too. If you disable referrer logging, then the browser always leaves the referrer blank. It's considered a privacy option, like enabling/disabling cookies.

[encyclo]

I always have referers disabled in normal use, but I activated them for the above test. The redirect process used by MSN Search does not pass the referer whatever settings you have.

[jomaxx]

I do see a handful of Netscape/Firefox referrers from MSN (about 0.5% of Explorer traffic).

Suspicious that the low number of Firefox users might be due to selection bias, I tested it myself. Sure enough, the referrer is suppressed when coming in from MSN, but not when coming from Google.

[bird]

I see some Gecko requests with a MSN referrer, but they are rare.

The JavaScript that makes use of the qping attribute should not prevent a referrer from getting sent. If it does, then that would be a bug in Firefox.

[ronburk]

I see some Gecko requests with a MSN referrer, but they are rare.

Likewise. Can you supply any of the exact UA fields? I got none in September and a grand total of 3 in August:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060719 Firefox/1.5.0.5

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1b1) Gecko/20060710 Firefox/2.0b1

And yet, I was able to immediately produce a hit with a Referer field from MSN search myself:

Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6

I guess it's possible these results are accurate for my site. About 2/3 of my total page hits from Gecko browsers are claiming revision 1.8.0.6. Hmm.

[vincevincevince]

I'm getting no referrer the first time I click, but I'm getting a proper search.msn.com referrer after going back and then clicking again.

"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6"

I tested search.msn.com, search.msn.co.uk and search.msn.com.my and all of them are the same.

[jomaxx]

It's an interesting effect, but I don't see it having much significance. It might go some ways towards explaining why my MSN referrers are so much lower than the equivalent period last year, though.

I'm more annoyed by the fact that Firefox is incompatible with the MSN-based official website for Rock Star: Supernova (a guilty pleasure). You'd think that, after ten years of this, these companies would have grown up and stopped victimizing the public by deliberately creating broken websites that force you to switch browsers or upgrade your plug-ins.

[br4inwash3r]

here's what i'm seeing on my stats

search.msn.com :
- Internet Explorer 87%
- Firefox 13%

search.msn.co.id :
- Internet Explorer 100%

search.msn.com.my :
- Internet Explorer 100%

[giuliorapetti]

I now get no referrals at all from MSN, may be due to the launch of the new "Live" engine.

I'm sure my ranking is still good even on the new engine, but very few referrals from msn.com or search.msn.com in the last 2 or 3 days.

Today, absolutely zero.

and: I also see the code

<a href="http://www.example.com/" gping="/GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&&DI=287&IG=088748d6etc.">

Surely this is connected to the g.msn.com/9SE/1?http://www.domain.com that we have been seeign in the logs in the last few months.

I guess this is not technically "a cookie" but "a way to track this search and this click" in the neural approach that MSN seems to be using 100% now...

1st consequence: yes, MSN has the right to do whatever they want, but the trackign of live.com searches is totally unpredictable

2nd consequence: I'm waiting for my ranking to increase soon as I'm deeply convinced that with the neaural component our sites will gain positions :-)

[NemoNemo]

Since 14-Sep i'm seeing 80% drop in msn refered visits. In parallel, there is 80% increase in blind referals. I looked more closely in the logs and those blind referals are almost exclusively IE browsers. The browsers that do show referals are now almost all non-IE. The javascript msn uses is not passing referal info.

[codegal]

Same thing here, no referrals from MSN. Found out after some testing that MSN was not passing the referral strings. It seemed to start happening after the 13th of September. It just shows direct access now. Darn! Thought I was getting more repeat visitors. Oh Well.

[wmuser]

Thanks for good info

[adamas]

Seems shortsighted to me.

A good way to get webmasters to recommend other search engines in their own selfinterest.

[ratebeer]

I'm all zeroed out as well. Looks like it was ebbing Wednesday and Thursday and pretty much zeroed out on Friday the 15th of September 2006.

Anyone with any work arounds? Is there a way to grab any information with ASPX or ASP from the browser history?

[Brett_Tabke]

Cudos MSN if this is not an accident - which it sounds like it is. If it were not - then this is the bravest act by a search engine I have seen since Overture fessed up its search keyword to the public.

There is a post around here somewhere from about 3 years ago where I proposed that SEO would be finished if engines switched from GET to POST.

Anyone know the url? It was something like "seo, three chars from done" or something like that. Erm, was it in supporters?

[Cluttermeleon]

OK, so what we are saying here is that unless the images are switched off in the developer toolbar (and let's face it, the vast majority of regular searchers won't have this functionality) then there is absolutely no way in the world we can track visitors from MSN?

Anyone?

[RonPK]

Coming soon, I guess: the MSN webmaster program. "Want to see your keyword stats? Join up! Enter your Windows Live ID here [..]"

@Cluttermeleon: yep. Also, referers will show if the user has JavaScript disabled.

[Brett_Tabke]

If all the engines would switch from GET to POST (thus eliminating referers), they would kill 50-90% of the page-optimization spam that hits their engines today.

However, lets be honest, they don't have the guts to do it on the organic side (too much value in the self promotion involved) and they also need to be able to have advertisers see those referrals on the paid side.

[RonPK]

In a widely used browser its very annoying to browse back to a POST page, so I don't see SEs using POST soon.

Anyway, in another thread [webmasterworld.com], MSNdude says about the missing referrer: "This is NOT by design. We're looking into it. Thanks for telling us."

[Brett_Tabke]

This was the long thread about Get vs Post and search engine referrals.
[webmasterworld.com...]

If you are a supporter:
[webmasterworld.com...]

[FanThomas]

The point is they are redirecting to new page in .js like this:

window.location = _pendingAddress;

so it depends on how the browser handles this command, in most cases no HTTP_REFERER is set I suppose

they are handling opening the page in new window separately, in such case the HTTP_REFERER should be passed

[tedster]

This reminds me of the problems that Looksmart had in the old days -- before they made that desparate money grab. A LookSmart directory purchase in those days did drive lots of traffic, but it mostly came through Looksmart partner sites. Looking at the referer in your logs, there was no way for the average webmaster (or an off-the-shelf stats package) to see what results they were getting for their LookSmart dollars.

It's not enough to be doing something for another business. In order for your to thrive, the other business must also KNOW you are doing it. And webmasters tend to be heavy influencers of tech use among their circle of contact. That's how Google got off the ground -- the tecchies spread the word. Live could eventually spread through the entire culture of web users in the same way, but only if webmasters actually see the results.

I'm sure MSN will not let this stand.

[Cluttermeleon]

Hey I started recording MSN traffic again today. Anyone else out there also recording traffic or am I a freak?