Way to "block" userAgents with ASP.NET

Just a little script I have written..

[kewlbeezer]

Here is the way our website "blocks" access for bad userAgent strings... Note that this isn't perfect, but if you asp.net on your machine, this works pretty well. badBots string on the blacklist.ascx page is the comma delimited "start" of the userAgent that is bad... ie, instead of having a separate entry for Educate Search V6B, Educate Search V32B, etc, I have an entry for "Educate Search"...

THis will only help on sites that only use ASP.NET

I hope this helps someone. I launched it so far and it works well. I have an email function that also gets called when a bad bot goes to our site, so I know that they tried to visit. I actively track all visits on the site, so I can then see if they found a way in (usually not). So this is pretty successful. My next project is to make this add the bad bots dynamically... probably into a textfile using a badBot trap. We'll see :)

Scott

== START page blacklist.ascx... ==

<script language="VB" runat="server">
public badBots as string = "Mozilla/3.0 (compatible; Indy Library),CBrowse,DBrowse,DDemo,Demo Bot,DSurf15a,EBrowse,Educate Search,Franklin Locator,FSurf15a,Full Web Bot,Industry Program,Mac Finder,Missauga Locate,Missigua Locator,PBrowse,PEval,Production Bot,PSurf15a,RBrowse,RSurf15a,SSurf15a"

private function get_ua_status()
dim status as boolean = false
dim ua as string = Request.UserAgent
dim badBotsArray as array = badBots.split(",")
dim badBotsSize as integer = uBound(badBotsArray)
dim i as integer = 0

do while (i <= badBotsSize)
if (left(ua, len(badBotsArray(i))) = badBotsArray(i)) then
status = true
exit do
end if

i = i + 1
loop

return status
end function
</script>

== end blacklist.ascx ==

== start global.asax ==

<!--#include file="blacklist.ascx"-->

<script language="VB" runat="server">
Sub Session_start(sender as object, e as eventargs)
dim status as boolean = get_ua_status()

if (status = true) then
response.redirect([Blacklist Page URL goes here])
end if
End Sub
</script>

== end global.asax ==

[Dreamquick]

Why not use regular expressions as they give you a far greater range of pattern matching (not just starts with) - also you could have a single string and then test it there and then without the need to split it before you can use it...

Just a thought from someone who's built something similar in classic ASP

- Tony

[kewlbeezer]

Well... Since I rarely use Regular Expressions, that's why I did the string that way...

Want to provide an example?

[carfac]

I would also recommend using regular expressions.... that way you do not have to re-enter a whole new UA when it changes versions and such. You could laso pick out those 'bots with "email" ANYWHERE in their name and exclude those.

Sorry, I do not know ASP, but I use Red Expressions in my apache, and have found great flexibility doing so!

dave

[wilderness]

RegEx+asp on a goggle resulted in 19,600 hits.

[Dreamquick]

Here's some old style ASP - it's just a quick concept, and remember that you need to turn the ¦ into solid vertical bars before trying the code on account of the forum changing them.

It simply grabs the current user-agent and runs it against the regular expression I've put into the constant, and if you happen to be using either Opera, Lynx or the Google proxy (or know how to fake it) then it prints some additional text.

I picked these three because they happened to cover both full patterns+wildcards as well as partial suffix and prefix patterns.

Another plus is that the basic content of all regular expressions are inter-operable rather than the MS version being incompatible with everything else.

Function TestRegExp( ByVal sInput, ByVal sRegExp )
'Code to evaluate a regular expression using the regexp object
Dim objRegular

TestRegExp = False

Set objRegular = New RegExp

objRegular.IgnoreCase = True

On Error Resume Next

objRegular.Pattern = sRegExp
TestRegExp = objRegular.Test( sInput )

If Err.Number <> 0 Then
Response.Write "REGEXP ::" & sRegExp
Response.Write "INPUT ::" & sInput
End If

On Error Goto 0

Set objRegular = Nothing

End Function

Const csSECrawlers = "^Opera/\d\.\d*\s\(.+;.*\)\s*.*$¦^Lynx/\d\.\d+¦\s\(Google\s(WAP¦CHTML)\sProxy/\d\.\d\)$"

Response.Write "<p>***START***</p>"
If TestRegExp( Request.ServerVariables("HTTP_USER_AGENT"), csSECrawlers ) Then
Response.Write "you're either on opera, lynx or using the google mobile proxy!"
End If
Response.Write "<p>***END***</p>"