Welcome to WebmasterWorld Guest from 23.22.182.29

Forum Moderators: phranque

Message Too Old, No Replies

Lost referrer

Loosing HTTP referrer in FireFox when user is coming from MSN

     
3:34 pm on Sep 8, 2006 (gmt 0)

New User

10+ Year Member

joined:Mar 14, 2005
posts:14
votes: 0


We are having a very odd issue. When a user is using a Mozilla browser and visits our site from MSN or from the Yahoo Site Explorer, the HTTP Referrer is empty. IE and Opera work fine but FireFox, Flock and Netscape don't.

We have tested this on multiple machines. The same machines visiting the site from Google or Yahoo search have the referrer.

Anyone else seeing this?

6:21 pm on Sept 9, 2006 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9063
votes: 2


That's an interesting observation - it appears to be to do with onclick events which remove the referer details during the subsequent redirect. Looking at MSN Search specifically, each site link is overridden by a Javascript function calling "GLinkPing.aspx":

<a href="http://www.example.com/" gping="/GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&amp;&amp;DI=293&amp;
IG=aeda64ef9eb548638565d31309e0e1f1&amp;POS=1&amp;CM=WPU&amp;CE=6&amp;
CS=AWP&amp;SR=6&amp;sample=0">example</a></h3>

(Carriage returns added for clarity, some details obfuscated)

If you enable referers in the browser and click the link, you can record the HTTP headers. You see something like this:

http://search.msn.com/GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&&DI=293
&IG=aeda64be9eb548638565d31309e0e1f1&POS=1&CM=WPU
&CE=6&CS=AWP&SR=6&sample=0

GET /GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&&DI=293
&IG=aeda64be9eb548638565d31309e0e1f1&POS=1
&CM=WPU&CE=6&CS=AWP&SR=6&sample=0 HTTP/1.1
Host: search.msn.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
Accept: image/png,*/*;q=0.5
Accept-Language: en-ca,en;q=0.8,fr-ca;q=0.5,fr;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
[b]Referer: ht[/b][b]tp://search.msn.com/results.aspx?q=example&FORM=QBHP[/b]
Cookie: [i](removed)[/i]

[b]HTTP/1.x 200 OK[/b]
Content-Length: 42
[b]Content-Type: image/gif[/b]
X-Powered-By: ASP.NET
P3P: CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND", policyref="http://privacy.msn.com/w3c/p3p.xml"
X-TraceID: b1f123f02fac4849968f6212fef11b81
Expires: Sat, 09 Sep 2006 17:52:06 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sat, 09 Sep 2006 17:52:06 GMT
Connection: keep-alive

----------------------------------------------------------

http://www.example.com/

GET / HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.5) Gecko/20060731 Ubuntu/dapper-security Firefox/1.5.0.5
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-ca,en;q=0.8,fr-ca;q=0.5,fr;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

HTTP/1.x 200 OK
Date: Sat, 09 Sep 2006 17:52:07 GMT
Server: Apache
Last-Modified: Sun, 03 Sep 2006 21:17:51 GMT
Etag: "2860310-bca-44fb467f"
Accept-Ranges: bytes
Content-Length: 3018
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: text/html

At the moment, I'm not entirely sure what the GLinkPing.aspx script is doing, but it declaring a MIME type of

image/gif
- it is actually a 1x1 pixel transparent GIF image rather than a HTML document. A tracking beacon, maybe, which remains in the browser cache? As you can see, the
gping
attribute is working as an onclick event, overriding the link. Once the image script is fetched, there is a redirect to the chosen website - but in Firefox (and thus in other Gecko-based browsers), the referer information is not included in the second request.

As there is no 301 or 302 redirect (only a 200 OK for GLinkPing.aspx), perhaps Gecko doesn't realise that it should pass on the information to the landing page? Perhaps the image MIME type is confusing things?

Does anyone get any referers from MSN Search from Gecko browsers (Firefox, etc.)? If referers are never passed, then we may be underestimating the number of visitors coming from MSN - even if you consider the probability that most MSN Search users are connecting with Internet Explorer.

4:25 am on Sept 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


I just ran filters over a couple hundred logs from June, to present -- and it seems to be true. I cannot find a Gecko browser with a referer of search.msn.com
2:15 pm on Sept 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 13, 2005
posts:1077
votes: 0


leave it to Microsoft to add in their own, custom, HREF attribute...

kaz

3:34 pm on Sept 10, 2006 (gmt 0)

Full Member

10+ Year Member

joined:May 1, 2002
posts:319
votes: 0


well. this is true. good to know at least.
4:21 pm on Sept 10, 2006 (gmt 0)

Full Member

10+ Year Member

joined:May 14, 2001
posts:262
votes: 0


I haven't checked lately but as of a few weeks ago my own clicks to my sites from msn and using firefox were just fine. I won't say that I know it is in the logs but my asp tracking picked up the referrer just fine.

I don't see a problem...

kaz

4:33 pm on Sept 10, 2006 (gmt 0)

Full Member

10+ Year Member

joined:May 1, 2002
posts:319
votes: 0


You should check now bmcgee, i am not sure yet when ... but currently ...
6:45 pm on Sept 10, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 19, 2004
posts:394
votes: 0


Opera has an option to enable/disable referrer logging. I wouldn't be surprised if FireFox does too. If you disable referrer logging, then the browser always leaves the referrer blank. It's considered a privacy option, like enabling/disabling cookies.
6:50 pm on Sept 10, 2006 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9063
votes: 2


I always have referers disabled in normal use, but I activated them for the above test. The redirect process used by MSN Search does not pass the referer whatever settings you have.
6:53 pm on Sept 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 6, 2002
posts:4768
votes: 0


I do see a handful of Netscape/Firefox referrers from MSN (about 0.5% of Explorer traffic).

Suspicious that the low number of Firefox users might be due to selection bias, I tested it myself. Sure enough, the referrer is suppressed when coming in from MSN, but not when coming from Google.

10:37 pm on Sept 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 10, 2001
posts:1550
votes: 10


I see some Gecko requests with a MSN referrer, but they are rare.

The JavaScript that makes use of the qping attribute should not prevent a referrer from getting sent. If it does, then that would be a bug in Firefox.

11:14 pm on Sept 10, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Oct 16, 2003
posts:971
votes: 0


I see some Gecko requests with a MSN referrer, but they are rare.

Likewise. Can you supply any of the exact UA fields? I got none in September and a grand total of 3 in August:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.5) Gecko/20060719 Firefox/1.5.0.5

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1b1) Gecko/20060710 Firefox/2.0b1

And yet, I was able to immediately produce a hit with a Referer field from MSN search myself:

Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6

I guess it's possible these results are accurate for my site. About 2/3 of my total page hits from Gecko browsers are claiming revision 1.8.0.6. Hmm.

2:27 am on Sept 11, 2006 (gmt 0)

Senior Member from MY 

WebmasterWorld Senior Member vincevincevince is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 1, 2003
posts:4847
votes: 0


I'm getting no referrer the first time I click, but I'm getting a proper search.msn.com referrer after going back and then clicking again.

"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.6) Gecko/20060728 Firefox/1.5.0.6"

I tested search.msn.com, search.msn.co.uk and search.msn.com.my and all of them are the same.

3:48 am on Sept 11, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member jomaxx is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Nov 6, 2002
posts:4768
votes: 0


It's an interesting effect, but I don't see it having much significance. It might go some ways towards explaining why my MSN referrers are so much lower than the equivalent period last year, though.

I'm more annoyed by the fact that Firefox is incompatible with the MSN-based official website for Rock Star: Supernova (a guilty pleasure). You'd think that, after ten years of this, these companies would have grown up and stopped victimizing the public by deliberately creating broken websites that force you to switch browsers or upgrade your plug-ins.

7:48 pm on Sept 13, 2006 (gmt 0)

New User

5+ Year Member

joined:Mar 31, 2006
posts:13
votes: 0


here's what i'm seeing on my stats

search.msn.com :
- Internet Explorer 87%
- Firefox 13%

search.msn.co.id :
- Internet Explorer 100%

search.msn.com.my :
- Internet Explorer 100%

5:22 pm on Sept 16, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Oct 29, 2005
posts:45
votes: 0


I now get no referrals at all from MSN, may be due to the launch of the new "Live" engine.

I'm sure my ranking is still good even on the new engine, but very few referrals from msn.com or search.msn.com in the last 2 or 3 days.

Today, absolutely zero.

and: I also see the code

<a href="http://www.example.com/" gping="/GLinkPing.aspx?/_1_9SE/1?http://www.example.com/&&DI=287&IG=088748d6etc.">

Surely this is connected to the g.msn.com/9SE/1?http://www.domain.com that we have been seeign in the logs in the last few months.

I guess this is not technically "a cookie" but "a way to track this search and this click" in the neural approach that MSN seems to be using 100% now...

1st consequence: yes, MSN has the right to do whatever they want, but the trackign of live.com searches is totally unpredictable

2nd consequence: I'm waiting for my ranking to increase soon as I'm deeply convinced that with the neaural component our sites will gain positions :-)

10:15 pm on Sept 16, 2006 (gmt 0)

New User

10+ Year Member

joined:May 20, 2005
posts:12
votes: 0


Since 14-Sep i'm seeing 80% drop in msn refered visits. In parallel, there is 80% increase in blind referals. I looked more closely in the logs and those blind referals are almost exclusively IE browsers. The browsers that do show referals are now almost all non-IE. The javascript msn uses is not passing referal info.
4:17 pm on Sept 17, 2006 (gmt 0)

New User from US 

10+ Year Member

joined:Oct 11, 2005
posts:38
votes: 0


Same thing here, no referrals from MSN. Found out after some testing that MSN was not passing the referral strings. It seemed to start happening after the 13th of September. It just shows direct access now. Darn! Thought I was getting more repeat visitors. Oh Well.
9:14 pm on Sept 17, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Apr 25, 2005
posts:677
votes: 0


Thanks for good info
1:30 pm on Sept 18, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Dec 30, 2002
posts:152
votes: 0


Seems shortsighted to me.

A good way to get webmasters to recommend other search engines in their own selfinterest.

9:38 pm on Sept 18, 2006 (gmt 0)

New User

10+ Year Member

joined:Oct 22, 2004
posts:31
votes: 0


I'm all zeroed out as well. Looks like it was ebbing Wednesday and Thursday and pretty much zeroed out on Friday the 15th of September 2006.

Anyone with any work arounds? Is there a way to grab any information with ASPX or ASP from the browser history?

2:14 am on Sept 20, 2006 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


Cudos MSN if this is not an accident - which it sounds like it is. If it were not - then this is the bravest act by a search engine I have seen since Overture fessed up its search keyword to the public.

There is a post around here somewhere from about 3 years ago where I proposed that SEO would be finished if engines switched from GET to POST.

Anyone know the url? It was something like "seo, three chars from done" or something like that. Erm, was it in supporters?

2:59 pm on Sept 20, 2006 (gmt 0)

New User

5+ Year Member

joined:Aug 31, 2006
posts:22
votes: 0


OK, so what we are saying here is that unless the images are switched off in the developer toolbar (and let's face it, the vast majority of regular searchers won't have this functionality) then there is absolutely no way in the world we can track visitors from MSN?

Anyone?

4:04 pm on Sept 20, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 22, 2002
posts:1749
votes: 0


Coming soon, I guess: the MSN webmaster program. "Want to see your keyword stats? Join up! Enter your Windows Live ID here [..]"

@Cluttermeleon: yep. Also, referers will show if the user has JavaScript disabled.

4:13 pm on Sept 20, 2006 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11



If all the engines would switch from GET to POST (thus eliminating referers), they would kill 50-90% of the page-optimization spam that hits their engines today.

However, lets be honest, they don't have the guts to do it on the organic side (too much value in the self promotion involved) and they also need to be able to have advertisers see those referrals on the paid side.

4:26 pm on Sept 20, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 22, 2002
posts:1749
votes: 0


In a widely used browser its very annoying to browse back to a POST page, so I don't see SEs using POST soon.

Anyway, in another thread [webmasterworld.com], MSNdude says about the missing referrer: "This is NOT by design. We're looking into it. Thanks for telling us."

8:31 pm on Sept 20, 2006 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38047
votes: 11


This was the long thread about Get vs Post and search engine referrals.
[webmasterworld.com...]

If you are a supporter:
[webmasterworld.com...]

8:46 pm on Sept 20, 2006 (gmt 0)

New User

10+ Year Member

joined:Sept 26, 2003
posts:6
votes: 0


The point is they are redirecting to new page in .js like this:

window.location = _pendingAddress;

so it depends on how the browser handles this command, in most cases no HTTP_REFERER is set I suppose

they are handling opening the page in new window separately, in such case the HTTP_REFERER should be passed

6:48 am on Sept 21, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:May 26, 2000
posts:37301
votes: 0


This reminds me of the problems that Looksmart had in the old days -- before they made that desparate money grab. A LookSmart directory purchase in those days did drive lots of traffic, but it mostly came through Looksmart partner sites. Looking at the referer in your logs, there was no way for the average webmaster (or an off-the-shelf stats package) to see what results they were getting for their LookSmart dollars.

It's not enough to be doing something for another business. In order for your to thrive, the other business must also KNOW you are doing it. And webmasters tend to be heavy influencers of tech use among their circle of contact. That's how Google got off the ground -- the tecchies spread the word. Live could eventually spread through the entire culture of web users in the same way, but only if webmasters actually see the results.

I'm sure MSN will not let this stand.

11:03 am on Sept 21, 2006 (gmt 0)

New User

5+ Year Member

joined:Aug 31, 2006
posts:22
votes: 0


Hey I started recording MSN traffic again today. Anyone else out there also recording traffic or am I a freak?
This 31 message thread spans 2 pages: 31
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members