Forum Moderators: coopster
I have found the "close to perfect .htaccess ban list" thread invaluable to determining friend or foe when it came to determining what user-agents I should allow unrestricted access to my site.
I started with the free block.php I found elsewhere, and added a "known browsers" list. Now I get emails everytime a banned browser accesses my site, as well as anytime I am accessed by an unknown browser. Access from "blank" user-agents is completely blocked.
-- some of you will notice some strange ua's in the "known" list; that's because I get a lot of accesses from WAP browsers --
Anyway, I can see that it wouldn't be too difficult to dress this thing up a little, maybe put the known and ban lists into text-files (lower overhead than DB). The next step could be allowing trusted sites to share the ban-list, or allowing a limited number of subscribers...
Another thing I was thinking about to automate the "known" user-agent issue, is to simply prompt unknown user-agents with a graphical code a la yahoo and ask the user to read and enter that code in order to receive access. Once that is complete, this user-agent would then be considered "known" and because a human response was received.
Would appreciate input.
<?php
$browser = array( "Wget", "EmailSiphon", "WebZIP","MSProxy/2.0", "EmailWolf","webbandit","MS FrontPage",
"GetRight", "AdMuncher", "Sqworm", "SurveyBot", "TurnitinBot", "WebMirror", "WebMiner",
"WebStripper", "WebSauger", "WebReaper", "WebSite eXtractor", "Teleport Pro",
"CherryPicker", "Crescent Internet ToolPak", "EmailCollect", "ExtractorPro",
"NEWT ActiveX", "sexsearcher", "ia_archive", "NameCrawler", "Email Spider",
"GetSmart", "Grabber", "GrabNet", "EmailHarvest", "Go!Zilla", "LeechFTP", "Vampire",
"SmartDownload", "Sucker", "SuperHTTP", "Collector", "Zeus", "Telesoft", "URLBlaze",
"VobSub", "Vacuum", "Space Bison", "WinWAP", "3D-FTP", "Wapalizer", "DTS Agent", "DA 5.",
"NetAnts", "NetSpider", "Disco Pump", "WebFetch", "DiscoFinder", "NetZip",
"Express WebPictures", "Download Demon", "eCatch", "WebAuto", "Offline Expl",
"HTTrack", "Mass Download", "Mister Pix", "SuperBot", "WebCopier", "FlashGet",
"larbin", "SiteSnagger", "FlashGet", "NPBot", "Kontiki" );
$known = array( "compatible; MSIE", "T720", "MIDP-1.0", "AU-MIC", "UP.Browser", "SonyEricsson",
"MobilePhone SCP", "NW.Browser", "Mozilla", "UP.Link", "Windows-Media-Player", "MOT-TA02",
"Nokia", "Opera/7", "NSPlayer", "GoogleBot/2", "Opera/6", "Panasonic", "Thinflow", "contype",
"klondike", "UPG1", "SEC-SGHS100", "Scooter", "almaden.ibm.com",
"SpaceBison/0.01 [fu] (Win67; X; ShonenKnife)", "Internetseer" );
$ua = strtoupper( $HTTP_USER_AGENT );
if ( $ua == "" )
{ $punish = 1; }
else
{ $punish = 0;
while( list( $key, $val ) = each( $browser ) )
{
if( strstr( $ua, strtoupper( $val ) ) )
{ $punish = 1; }
}
}
$notknown = 1;
while( list( $key, $val ) = each( $known ) )
{ if( strstr( $ua, strtoupper( $val ) ) )
{ $notknown = 0; }
}
if( $punish )
{
$msg = "The following session generated banned browser agent errors:\n\n";
$msg .= "Remote Addr: $REMOTE_ADDR\n";
$msg .= "Client IP: $HTTP_CLIENT_IP\n";
$msg .= "Agent: $HTTP_USER_AGENT\n";
$msg .= "Referrer: $HTTP_REFERER\n";
$msg .= "Document: $SERVER_NAME" . $REQUEST_URI . "\n";
$headers .= "X-Priority: 1\n";
$headers .= "From: BanBot <webmaster@domain>\n";
$headers .= "X-Sender: <webmaster@domain>\n";
mail( "webmaster@domain", "BANBOT: $HTTP_USER_AGENT", $msg, $headers );
echo "<HTML><head><title>Access Denied</title></head>
<p>We're sorry. The software you are using to access our website is not allowed.
Some examples of this are e-mail harvesting programs and programs that will
copy websites to your hard drive. If you feel you have gotten this message
in error, please send an e-mail to the webmaster. Your I.P. address has been logged.</p>
</body></HTML>";
exit;
}
if( $notknown && ( $uacheck<>1 ) )
{ SetCookie( "uacheck", "1", time() + 172800 );
$msg = "The following session generated an unknown browser agent exception:\n\n";
$msg .= "Remote Addr: $REMOTE_ADDR\n";
$msg .= "Client IP: $HTTP_CLIENT_IP\n";
$msg .= "Agent: $HTTP_USER_AGENT\n";
$msg .= "Referrer: $HTTP_REFERER\n";
$msg .= "Document: $SERVER_NAME" . $REQUEST_URI . "\n";
$headers .= "X-Priority: 1\n";
$headers .= "From: BanBot <webmaster@domain>\n";
$headers .= "X-Sender: <webmaster@domain>\n";
mail( "webmaster@domain", "UNKNOWN: $HTTP_USER_AGENT", $msg, $headers );
}
?>
i'd like admit one thing which tends to be well working in practise, for both email harvesters and site mirroring tools. the principle is quite easy:
you place a link into your page(s) code that leads to a pit. if this pit is requested, the requesting ip-address is blocked for a specific amount of time.
the link itself is hidden on the normal page, so that no human user would click on it, ie a one pixel image with no border and no alt-text.
you'll need a blacklist in a db or in a plain file for this, so in cases of overhead that might not be suitable, but if you're in a real concern about leechers and harvesters, this might be the right, ip-based solution.
professional harvester programs can hide/trick completely by simulating a browser. a pit is a trap which can trick those.
added:
to check your page against those, take a look at the freeware tool called webcow [subfiles.net]. if this tool can pick your files, any bot could.
GoogleBot has visited twice, and when it comes back, I don't want to set a trap for it...
As for testing, I've been playing with www.wannabrowser.com to pull my site and see what my script does.
Because legitimate spiders follow the robots.txt convention, they will not fall into the pit.
Jim
the yahoo-like idea, to enter a the text on a graphic might be hard for wap browser. sometimes ago i had the idea that a simple mathematical question can be displayed and the user has to calculate the result and enter it. the math question can be created randomly and (maybe) displayed with images (image recognition is needed then to override), but maybe it's even not a need.
example:
5 * 2 + 2 = [_________]
this can even work with lynx (if you don't take any graphics).
i disliked the yahoo thing (or altavista) because it was too hard to read.
i think it's a nice idea to do it this way to get more info about user-agent strings, too.
maybe my idea is usefull, feedback appreciated.
By the way, I've been tracking visiting UAs and their accept headers for the past week, and have a list of over 1,000 UAs with headers if anyone wants them.
I think math-problems may be a good method too, but I'd be concerned that if people re-use that script, badbots will get smart enough to parse and calculate it (there's gotta be a reason why yahoo went to the graphical approach). Maybe there can be a good medium by displaying math problems graphically...?
<note to self>It may be time to provide an updated (and more sophisticated version) of trap.pl</note>
P.S. block.php looks suprisingly similar to a version of ban_bot.cgi [webmasterworld.com] my good friend Froggyman posted here on WebmasterWorld a couple of years ago. You know Froggyman is a really special guy 'cuz he hasn't posted in nearly 2 years and he's still a full member. :)
I think a block.php type of solution is best for webmasters who do not have access to a .htaccess file. A .htaccess file is much faster and you would free up server resources. With .htaccess capability, you could still use the script on a custom 403 error page allowing you to receive E-mail notifications. For example, EmailSiphon is banned via .htaccess and redirected to the 403 page which triggers the script to send the E-mail.
I couldn't help but notice, but was the term 'BanBot' already in that script when you downloaded it?
