Forum Moderators: coopster & phranque

Message Too Old, No Replies

Ban User Agents With Perl/SSI -- Pesky Bots Will Go Away

Heres how to do it.

         

Froggyman

4:16 am on Jun 9, 2001 (gmt 0)



A lot of people have been asking how to block an agent name so I decided to write my own. For those of us that can't block an Agent through .htaccess here is a simple (yet efficient) way of doing it with SSI.

Cut and paste this file and save as ban_bot.cgi and place in your cgi-bin.


#!/usr/local/bin/perl

#######################
# Browser Agents Banned
@browser = ("Wget/1.6","Zeus","EmailSiphon"); # List of Banned Agents - Add as many as you like

########################
# Get Browser Agent Info
$get_agent = $ENV{'HTTP_USER_AGENT'}; # Get Browser Agent - Requires SSI

########################
# Check Against Ban List
foreach $ban (@browser) {
if ($get_agent =~ /$ban/) {
$punish = 1;
}
}
if ($punish == 1) { # Banned agent is placed into an infinite loop
while (1) {
$x++;
}
}
else {
print "Content-type: text/html\015\012\015\012\n\n"; # The innocent are set free
}

############
# End Script

Place <!--#include virtual="/cgi-bin/ban_bot.cgi"--> at the top of each page on your site and that's it!

I've tested this with my best snooping software and it works great. You may freely use and modify this script as you see fit although I would ask that users post any improvements made so that the rest of us can benefit from them.

icehousedesigns

4:18 pm on Jun 25, 2001 (gmt 0)



FYI I was having trouble with the DATE_LOCAL. I wasn't getting the timestamp on e-mails... so I changed it around a bit..works fine now.

$date = scalar localtime ( time ); print MAIL "Time: $date \n"; 

Also had a problem with it displaying the full URL in the e-mail...

Changed:

print MAIL "Document: $ENV{'SERVER_NAME'}$ENV{'DOCUMENT_URI'}\n";

to:

print MAIL "Document: $ENV{'SERVER_NAME'}$ENV{'REQUEST_URI'}\n";

The following session generated a banned browser agent error:

Host: myipsnipped
Agent: EmailWolf 1.00
Referrer:
Document: www.mydomain.com/myurl/blahblah.php3
Time: Mon Jun 25 12:13:20 2001

------------------------------------------------------

:)

Froggyman

5:13 pm on Jun 25, 2001 (gmt 0)



EmailWolf huffed and puffed and then gave up :)

Some servers will use REQUEST_URI instead of DOCUMENT_URI. If one doesn't work, try the other.

I'm working on a new version of this script. Planned features: Ban agents or IP's and punish based on threat, redirect browsers to custom error document, add/remove agents or IP's from a list (instead of modifying/uploading the script each time), ban visitor based on referrer (e.g. [iaea.org...] redirect browsers based on search engine referral, and graph banned usage. I'll make it public once it's complete.

It's nice having a bit of control now, especially when there was no control before. :)

icehousedesigns

1:42 am on Jun 26, 2001 (gmt 0)



That would rock Froggyman. Those features would be incredible to have..I was going to play around a bit and make a switch for redirct to a specific URL..be sure you let us all know when its done :) I'm sure many more than I will be drooling in anticipation. :)

icehousedesigns

7:14 pm on Jun 28, 2001 (gmt 0)



With the help of our Admin here ( whos kindness has been helping me all along ) He re-wrote the script in PHP..it works awesome. Keep in mind you have to include either this version or the SSI version at the VERY top of your docs even before the html tag..or the server will return a 200 and will allow the document to be viewed.

This is called via the standard PHP include.


<?php include ("/server/path/to/block.php3"); ?>

begin block.php3


<?php
$browser = array ("Wget", "EmailSiphon", "WebZIP","MSProxy/2.0","EmailWolf","webbandit","MS FrontPage");

$punish = 0;
while (list ($key, $val) = each ($browser)) {
if (strstr ($HTTP_USER_AGENT, $val)) {
$punish = 1;
}
}

if ($punish) {
// Email the webmaster
$msg .= "The following session generated banned browser agent errors:\n";
$msg .= "Host: $REMOTE_ADDR\n";
$msg .= "Agent: $HTTP_USER_AGENT\n";
$msg .= "Referrer: $HTTP_REFERER\n";
$msg .= "Document: $SERVER_NAME" . $REQUEST_URI . "\n";
mail ("youremail@yourdomain.com", "BANNED BROWSER AGENT ERROR", $msg);

// Print custom page
echo "<HTML>
<head>
<title>Access Denied</title>

</head>

<p>We're sorry. The software you are using to access our website is not allowed.
Some examples of this are e-mail harvesting programs and programs that will
copy websites to your hard drive. If you feel you have gotten this message
in error, please send an e-mail addressed to admin. Thanks.</p>
<BR>
-Your name.
<BR>
</body>

</HTML>";
exit;
}

?>

LunaC

7:27 pm on Jul 5, 2001 (gmt 0)

10+ Year Member



kapow:
Do you know where I can get more detailed info how to do what was done on [julianhaight.com...] ?

That looks like it might be the perfect solution for one of my sites.

circuitjump

7:22 pm on Jul 6, 2001 (gmt 0)

10+ Year Member



What if I want to redirect the agent to a page that gives them an access denied, how would I do it in perl and would it be a good idea?

Thanks

circuitjump

6:39 pm on Jul 9, 2001 (gmt 0)

10+ Year Member



I made the concept in ASP.
I figure since we got it in perl and PHP, might as well get it in ASP.

Here it is,

<%

' Declare browser array
Dim browser(6)

browser(0) = "Wget"
browser(1) = "EmailSiphon"
browser(2) = "WebZip"
browser(3) = "MSProxy/2.0"
browser(4) = "EmailWolf"
browser(5) = "webbandit"
browser(6) = "MS FrontPage"

' Declaring variables
Dim user_agent, host, document, referrer, punish

' Get User Agent Info
user_agent = Request.ServerVariables("HTTP_USER_AGENT")
host = Request.ServerVariables("REMOTE_ADDR")
document = Request.ServerVariables("SERVER_NAME")
referrer = Request.ServerVariables("HTTP_REFERER")

' Check Agent

punish = 0

If Left(user_agent, 4) = browser(0) Then
punish = 1

ElseIf Left(user_agent, 11) = browser(1) Then
punish = 1

ElseIf Left(user_agent, 6) = browser(2) Then
punish = 1

ElseIf Left(user_agent, 11) = browser(3) Then
punish = 1

ElseIf Left(user_agent, 9) = browser(4) Then
punish = 1

ElseIf Left(user_agent, 9) = browser(5) Then
punish = 1

ElseIf Left(user_agent, 12) = browser(6) Then
punish =1

End If

If punish = 1 Then

' Mail the Webmaster

Set objMail = Server.CreateObject("CDONTS.Newmail")
objMail.To = "rb3@redblue3.com"
objMail.From = "rb3@redblue3.com"
objMail.Subject = "BANNED BROWSER AGENT ERROR"
objMail.Body = "The following session generated banned browser agent errors: Host: " & host & " Agent: " & user_agent &_
" Referrer: " & referrer & " Document: " & document & "."
objMail.Send
Set objMail = Nothing

' Redirect to access denied page

Response.Redirect ("access-denied.html")
End If
%>

I'm still getting used to programming in ASP and PHP so some of this could most likely be done in a much more efficient way but I figure it's a start.

themoff

5:24 pm on Jul 15, 2001 (gmt 0)

10+ Year Member



In ASP, a neater way of checking the given UA against the list of banned UAs would be:

punish = 0
FOR index=0 TO UBOUND(browser)
IF Left(user_agent,LEN(browser(index)))=browser(index) THEN
punish=1
END IF
NEXT

Also, instead of redirecting to a new page, would this following code work?


Response.Status = "403 Forbidden - Your UA blows goats."
response.end

I don't fully understand the intracacies of the HTTP protocol, so apologies if I've got the wrong end of the stick here.

Cheers, Robin

circuitjump

1:41 pm on Jul 16, 2001 (gmt 0)

10+ Year Member



Actually that sounds great. I new that there was a better way of doing it in ASP.

themoff

2:59 pm on Jul 16, 2001 (gmt 0)

10+ Year Member




if ($punish == 1) { # Banned agent is put to sleep
sleep(60);

Is there an ASP equivilant of this sleep command?

circuitjump

11:00 pm on Jul 16, 2001 (gmt 0)

10+ Year Member



Not that I know of. I tried looking around and could not find anything.

Sorry

Key_Master

1:13 am on Jul 27, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



[hotscripts.com...]

RE: Line 3 in the download

# This script is freeware as long as this header remains intact

A reminder...this script is freeware regardless.

Key Master (Unretired my early '80s nick)
aka Froggyman

toolman

1:24 am on Jul 27, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I thought that was you over there at hotscripts froggyman...welcome back :)

Bolotomus

10:03 pm on Aug 2, 2001 (gmt 0)

10+ Year Member



How about saying something like...

if ($user_agent =~ /well-known-email-harvester/) {
for (1...5000) {
print &generate_bogus_email(),"<br>\n";
}
}

I figure if it's going to be incrementing some dumb counter for a while you might as well doing something productive with your clock cycles.

If it's email addies they want, it's email addies they get.... by the truckload! Mauahahahahaha

Key_Master

8:49 am on Sep 9, 2001 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



As promised BanBot is nearing completion. Currently I'm working on the log file analyzing portion of the script. See URL in profile for example (I went a little overboard- AXS might be in for a little competition ;)).

I'm looking for input on how to store the log file. If you had a choice, would most of you prefer a single log file or a new log file created automatically by the script for each day of the week? Is the latter option feasible for all servers?

This 45 message thread spans 2 pages: 45