HTTP-REFERER problem

[PhilC]

Hi,

I need to check the referer of site visitors and I can't get Perl's HTTP-REFERER to yield any information at all. I've even copied and pasted code from perl.com which lists all the $ENV variables and their contents. Everything seems to work ok but still HTTP-REFERER reveals nothing.

I know that the referer should have some content because I've tried it with clicks from search engine listings and the referers appear in the log. I just can't get hold of the data in HTTP-REFERER.

Another script that loops through and prints all the $ENV variables and their contents doesn't even list HTTP-REFERER.

Any help would be appreciated.

[PsychoTekk]

the examples from perl.com work fine i think.
i suppose the problem has less to do with the scripts but
maybe your browser does not support referer / referers are
disabled?
what browser do you use?

[chris_f]

This may sound supid but it's an easy mistake to make (and boy have I done it alot in the past).

You aren't testing you code by going directly to page are you? If so, you have no referrer. I would try linking to you page with the code from another page and preferably another domain.

Chris.

[PhilC]

Thanks for the extremely quick replies, guys.

PsychoTekk:
yup. I had no doubt about the perl.com scripts. But it isn't a browser problem. It's a serverside problem. The perl script runs on the server and it's there where I can't get hold of the referer.

chris_f:
no. I know not to expect a referer when going straight from my browser :) That's why I've been testing from search engine listings. Also, in case of a cache somewhere down the line (and there shouldn't be one with cgi scripts), I've left the script in place so that people searching the engines and clicking from there are actually testing it for me.

My script logs their user-agents and referers. I'm getting their user-agents ok but not their referers. It's simply empty - or in view of the $ENV loop that I tried at one point, the HTTP-REFERER variable doesn't even exist in the $ENV array.

[PsychoTekk]

well i'm not sure if i got the point now but
the referer does neither show up in your domain's logfile,
nor in the HTTP_REFERER $ENV variable...
one can configure the apache logfiles to in/exclude several
variables / to log them to different logfiles but that should
not affect the $ENV variables in any way.
i'm not sure what it means that the HTTP_REFERER var is not even
present...
has anybody here heard of the possibility to block variables serverside?
i know that one can do this with cacheservers - do you use
a gateway, an accelerator proxy or does your ISP use backbone cache servers?
you can check this by doing a HTTP request using the TRACE method instead of
HEAD or GET.

[PhilC]

yes. the referer does show in the logfile. I've run 2 different looping scripts to view all the $ENV variables and contents and they both omit the HTTP_REFERER element. So it looks like it doesn't exist in $ENV.

Any ideas how to alter some config or other on the server to include it?

[PsychoTekk]

the only thing i know of is to set/unset/append/add headers and their values
using the Header directive (httpd.conf/.htaccess)
but there might be other modules to do so as well

[PhilC]

I had a scan through the httpd.conf file for see if there were any relevant looking lines, but i didn't find any.

So I've come up with a possible solution which is to abandon Perl for this script and do it in PHP instead. All I have to do now is find out how to access the referer in PHP :)

Many thanks for taking the time to help.

[PsychoTekk]

i think in PHP it's something like
$HTTP_SERVER_VARS["HTTP_REFERER"]

[gsx]

P.S. A firewall will often block the referrer.

[PsychoTekk]

gsx, don't some firewalls encode the referer variable to
an all uppercase string in the weferer variable?
i'm wondering where those weferer headers come from since
i could not find a RFC

[gsx]

I'm not sure how the various firewalls work, some blcok the referrer by changing it something else such as '~~~~~~~', some apparently remove it altogether.

If PhilC is clicking on his own machine with a firewall running, this could be the effect of the firewall removing the information, thus his page shows no referrer. The code may be running perfectly and converting to PHP may make no difference. Worth checking before he goes to the trouble.

[PhilC]

Well, I tried it in PHP3. I listed all the environment variables and HTTP_REFERER is conspicuous by its absence - just as I got when listing them all with Perl.

It seems to me that the problem is on the server but I've no idea what's wrong. I'm taking steps to try to find out what's wrong but I'm not optimistic of a positive result. It would probably be better if I take the question to an Apache forum, I think.

Incidentally, gcx, I'm testing by clicking from search engine listings so the referer is passed - it appears in the log file. Also, I'm recording information in a file (as a test) when other people arrive from search engines, so any system that I am using doesn't enter into it. The file captures some info but not the referer.

Since HTTP_REFERER doesn't appear in the list of environment variables using both Perl and PHP3 loops, it must be a server config problem.

Thanks to all for trying to solve it for me :)