I have a trap script that emails me whenever somebody/thing triggers the trap. It sends me the intruder's IP, UA, Method, Host, Date, filename and path. I am trying to get it to also send me the referer, but when I include the line for that field it always causes a 500 server error. Interestingly enough, it still sends out the email report, even though it doesn't display the text message that is supposed to be displayed to the offender.
My script pertaining to just the email function is below:
<snip>print commands and ban script codes here</snip>$remreq = $ENV{REQUEST_URI};
$remaddr = $ENV{REMOTE_ADDR};
$usragnt = $ENV{HTTP_USER_AGENT};
$remmeth = $ENV{REQUEST_METHOD};
$remhost = $ENV{HTTP_HOST};
$referer = $ENV{HTTP_REFERER};
$date = scalar localtime(time);
open(MAIL, "¦/usr/sbin/sendmail -t") ¦¦ die "Content-type: text/text\n\nCan't open /usr/sbin/sendmail!";
print MAIL "To: xxx\@xxx\.xxx\n";
print MAIL "From: xxx\@xxx\.xxx\n";
print MAIL "Subject: You caught another one!\n";
print MAIL "The ip address: $remaddr was banned on $date \n";
print MAIL "The file requested was: $remreq\n";
print MAIL "The method used was: $remmeth\n";
print MAIL "The intruder's user agent was: $usragnt\n";
print MAIL "The remote host was: $remhost\n";
# The next line's referer variable causes a 500 server error
print MAIL "The referrer was: $referer\n";
close(MAIL);
exit;
TIA, Wiz
print MAIL "Subject: You caught another one!\n\n";
I can't see how that would cause the effect you're describing though...
Do you have access to the error log to find out what it's not happy with?
Thanks again, Wiz
#!/usr/bin/perl -w$rootdir = "$ENV{DOCUMENT_ROOT}";
$intruder_ip = $ENV{'REMOTE_ADDR'};
$intruder_ip =~ s/\./\\\./gi;
open(HTACCESS,"".$rootdir."/\.htaccess") ¦¦ die $!;
@htaccess = <HTACCESS>;
close(HTACCESS);
open(HTACCESS,">".$rootdir."/\.htaccess") ¦¦ die $!;
print HTACCESS "SetEnvIf Remote_Addr \^".$intruder_ip."\$ ban\n";
foreach $deny_ip (@htaccess) {
print HTACCESS $deny_ip;
}
close(HTACCESS);
sleep (10); # Keep them waiting after banning them
print "Content-type: text/html\n\n";
print "<html>\n";
print "<head>\n";
print "<title>FormMail Phisher Attitude Adjustment</title>\n";
print "<meta name=\"robots\" content=\"noindex,nofollow\">\n";
print "</head>\n";
print "<body><basefont face=\"arial,helvetica,sans serif\">\n";
print "<center><h1>BANNED!</h1></center>\n";
print "<h2>This script records IP addresses of FormMail script abusers for exclusion from our website</h2>\n";
print "</body>\n";
print "</html>\n";
$remreq = $ENV{REQUEST_URI};
$remaddr = $ENV{REMOTE_ADDR};
$usragnt = $ENV{HTTP_USER_AGENT};
$remmeth = $ENV{REQUEST_METHOD};
$remhost = $ENV{HTTP_HOST};
$referrer = $ENV{'HTTP_REFERER'};
$date = scalar localtime(time);
open(MAIL, "¦/usr/sbin/sendmail -t") ¦¦ die "Content-type: text/text\n\nCan't open /usr/sbin/sendmail!";
# print MAIL "To: xxx\@xxxxxx\.net\n";
print MAIL "To: xxx\@xxxxxx\.net\n";
print MAIL "From: xxx\@xxxxxxx\.net\n";
print MAIL "Subject: You caught another one!\n";
print MAIL "The ip address: $remaddr was banned on $date \n";
print MAIL "The file requested was: $remreq\n";
print MAIL "The method used was: $remmeth\n";
print MAIL "The intruder's user agent was: $usragnt\n";
print MAIL "The remote host was: $remhost\n";
print MAIL "The referrer was $referrer\n";
close(MAIL);
exit;
[edited by: sugarkane at 1:19 pm (utc) on Oct. 7, 2003]
I am removing that variable and the associated print line to keep the script usable, unless someone can show me a working solution for my situation.
Wiz
I did not have access to the server error logs to see what was making the Perl engine unhappy, so I used an idea I got while Googling on the keywords "errors env http_referer." Many other people have been experiencing similar troubles with this variable and the best replies involved testing the server for the absence or presence of this variable and writing an IF statement to handle both situations. I found a simple server ENV report script and ran it, whereupon I discovered the absence of any HTTP_REFERER field with a directly typed request for my script, but the existence of it if I created and followed a link from a test page. This absence of the requested variable was causing the Perl engine/server interaction to crap out (it does not print out the cause of the (500) server error, just the fact of it).
Here is the code I used for reporting the server variable present when the trap is executed:
#!/usr/bin/perl -wprint "Content-type: text/html\n\n";
print "<p>The Server Environment is:</p>\n";
foreach (sort keys %ENV)
{
print "$_=$ENV{$_}<BR>\n";
}
# IP capture and htaccess write codes here
# All desired server variables and screen print statements here
# print MAIL statements here
exit;
Here is the solution I have come up with after much experimenting:
if ( exists $ENV{'HTTP_REFERER'} )
{
$referer = $ENV{'HTTP_REFERER'}
}
else
{
$referer = "a direct request, or a cloaked or blank referrer"
}
Thanks to all who tried to help ... Wiz
Do you have a similar problem if no HTTP_USER_AGENT is available? I'm just wondering, because this sounds like a really odd problem...
That brings up a second question: IF the HTTP_REFERER is missing in a normal request, does your raw server access log show "" or "-" as the referrer? I'm wondering if your server setup is skipping the step where a non-existent referrer is detected, and the HTTP_REFERER variable is populated with a hyphen instead of showing blank. If so, this might be a strong clue for others who have the same problem in the future.
Jim
I will use Wannabrowser to check out your question about a blank UA field in the trap script and will edit this Post with the results.
Thanks for the input Jim.
Wiz
BTW: I wonder why the gurus of Perldom misspelled Referer, as in HTTP_REFERER? My dictionary says it should be Referrer. Que Sera, Sera.
Main Entry: re·fer
Pronunciation: ri-'f&r
Function: verb
Inflected Form(s): re·ferred; re·fer·ring
I can't get a blank UA or Referer from WannaBrowser.
I'll think of something else to try.
[edited by: Wizcrafts at 7:13 pm (utc) on Oct. 6, 2003]
And you can easly remove that option if you are running the script on the server. This is most of the times only used by the programmers for to trap errors and misspellings in the script.
About the misspelled 'Referer'.
Being a perl guru don't mean that you can spell.
They know it now but can not change it anymore... to many programs rely on it now :)
I think the spelling error predates PERL. It's built-in to Apache, and may even predate that! PERL just inherited that spelling because it is checking a predefined server environment variable.
In Apache, a blank referrer or UA is shown as "-" in the logs. However, when using RewriteCond %{HTTP_REFERER} or RewriteCond %{HTTP_USER_AGENT} to test for blanks, you'd use ^$ as the pattern to match. Then, some smart-alec harvester got the bright idea that he could use a hyphen for his UA and referrer, and so bypass the test for a blank. As a result, if I detect a blank referer and a blank UA, I block the access with a 403 response. However, if I detect a hyphen user-agent and a hyphen referer, I send the request on to our favorite script for a more permanent solution. You may be seeing the hyphen trick in your logs.
Jim
I finally got Wannabrowser to emit a blank UA by typing in ALT + 255. It is reported as nothing in the email and did not cause any server error. However,the User Agent field does appear in the printout of the server environment, so the extended characters did not really produce a blank UA and my test is!valid. I may alter my IE UA in the registry and try it again. That way I will have a true blank UA and can test the code that is supposed to ban dashes only, but pass blanks.
Wiz
I have also tested the script with the -w removed from the shebang and no errors were generated with a missing referrer, so that is an option to including IF tests.
Anon, here is the code to detect the absence or presence of a http_user_agent field:
<snip>
if ( exists $ENV{'HTTP_USER_AGENT'} )
{
$usragnt = $ENV{'HTTP_USER_AGENT'}
}
else
{
$usragnt = "cloaked or blank User Agent"
}
<snip>
print MAIL "The intruder's user agent was: $usragnt\n";
<snip>
close(MAIL);
Wiz
In both cases I couldn't replicate the problem.
Maybe it's a version issue on some combinations of OS, perl etc.
My dev server runs FreeBSD 3.1, Apache 1.3.4, Perl 5.005.03
BTW the mis-spelling of referer is an Apache legacy from a typo in an early build.
Beside that, where I live, if you want anbody to do something for you If is a very big word. therefore, using Ifs in my scripts just plain comes natural to me, don't cha know!
Wiz
I'm surprised no one has pointed out the most elegant solution to the missing HTTP_REFERER:
$referer = $ENV{'HTTP_REFERER'} ¦¦ "There is no referer!";
(Don't forget to fix the broken vertical pipes!)
Balam solution: 58 bytes
Wizcrafts solution: 141 bytes
;)
...and is forcing me to become a more precise code programmer.
You make it sound like a bad & dirty thing... ;)
To anyone in general...
Isn't removing the "-w" switch from your shebang line the perl equivalent of hiding your head in the sand? I mean, the errors are still there but you just don't want to see & deal with them... Out of sight, out of mind, ergo - no errors. :(
use strict? use warnings? Taint-checking? Huh?!
$referer = $ENV{'HTTP_REFERER'} ¦¦ "There is no referer!";
Thanks for that "compressed" line of code and scripting assistance! :)
Balam solution: 58 bytes
Wizcrafts solution: 141 bytes
use strict? use warnings? Taint-checking? Huh?!
Always learning, I remain, Wiz (in the school of hard knocks)
To anyone in general...Isn't removing the "-w" switch from your shebang line the perl equivalent of hiding your head in the sand? I mean, the errors are still there but you just don't want to see & deal with them... Out of sight, out of mind, ergo - no errors. :(
use strict? use warnings? Taint-checking? Huh?!
I use...
use strict;
use warnings FATAL => 'all';
no warnings 'redefine'; # to be remove in productionWhile programing for mod_perl, what is more difficult and strict than normal programing for perl (if you don't want problems with nested subroutines).
But putting that on a live server while visitors are using the script is asking about problems. Its for to help programers, not for to use live on the web server. Other problem is that it take server resources to check every time if the script validate or not (slow scripts).
The only problem is the the var $ENV{HTTP_REFERER} is not set and perl warn him for that, CAN be a spelling mistake.
HUH?!
balam has the best option from all.
$referer = $ENV{'HTTP_REFERER'} ¦¦ 'There is no spoon!';($ENV{'HTTP_REFERER'})? ($referer = $ENV{'HTTP_REFERER'}) : ($referer = 'There is no spoon!');
