Can Anyone Help Me Please.

Using the global.asa & regular expressions to detect bad bots and redirect.

[tantalus]

Hi

It would really be appreciated if someone could read through the following code and let me know whether I am on track or not. I’m not too familiar with the global.asa or regular expressions. (Can the RegExp object be used in the global.asa? for example)

Essentially, I want to use the Session_OnStart procedure of global.asa to
read a text file (badbots.txt) and compare the entries against the current user agent. If a match is found then I redirect to a blank html file.

This is what I have come up with so far, will this work….

<script language="vbscript" runat="server">

Private Sub Session_OnStart

Const Filename = "/badbot.txt"' file to read

Const ForReading = 1, ForWriting = 2, ForAppending = 3
Const TristateUseDefault = -2, TristateTrue = -1, TristateFalse = 0

' Get user agent
Dim UA
Set UA = Request.ServerVariables("HTTP_USER_AGENT")

' Create a filesystem object
Dim FSO
set FSO = server.createObject("Scripting.FileSystemObject")

' Map the logical path to the physical system path
Dim Filepath
Filepath = Server.MapPath(Filename)

if FSO.FileExists(Filepath) Then

' Get a handle to the file
Dim file
set file = FSO.GetFile(Filepath)

' Open the file
Dim TextStream
Set TextStream = file.OpenAsTextStream(ForReading, -2)

' Read the file line by line
Do While Not TextStream.AtEndOfStream
Dim Line
Line = TextStream.readline

' Create instance of RegExp object
Dim objRegExp
Set objRegExp = New RegExp

objRegExp.IgnoreCase = True
objRegExp.Pattern = Line

' Compare Line with UA (User Agent) using regular expressions from text file itself
Dim strStringToSearch
strStringToSearch = UA

' Returns true, if match found
Validate = objRegExp.Test(strStringToSearch)
If Validate = True Then
Set objRegExp = nothing
Set TextStream = nothing
Set FSO = nothing
Set UA = nothing
Session.Abandon
'Warning only redirect to a HTML page
Response.Redirect "/blank.htm"
Response.End
Else
'Clean up
Set objRegExp = nothing
End If
Loop

Set TextStream = nothing

End If

Set FSO = nothing
Set UA = nothing
End Sub

</script>

Here’s an example of the badbots.txt:

[*Larbin*]
[*NaverRobot*]
[*www4mail/*]
[AaronCarter/*]
[AgentName/0.1 libwww-perl/*]
[ah-ha.com crawler ([e-mail])]
[Alexibot*]
[AltoVistoWebCrawler]
[amphetameme crawler ([e-mail])]
[AOLserver-Tcl/*]
[appie 1.1 (www.walhello.com)]
[Aqua_Products*]
[ASPSeek*]
[asterias/*]
[ATA-Translation-Service]
[b2w/0.1]
[BackDoorBot/*]
[BitBeamer/*]
[BlowFish/*]
[Bookmark search tool*]
[BotALot*]
[BotRightHere*]
[Brouser]
[BuiltBotTough*]
[Bullseye/*]
[Bull**** browser (Hackers OS)]

Etc, etc.

Thanks in advance. (Please be gentle slow learner on board!)

[plumsauce]

I wouldn't do it because it will require
sessions to be turned on.

But, if you choose to proceed, a performance
tip would be to read the text file in at
application start to a application variable
and then refer to it from your session start.

This avoids the overhead of file access on
every session start.

[Dreamquick]

I have something like this already & running at the moment, as Plubsauce suggests loading the data every time isn't ideal so I chose to use an application variable to hold this data, reloading the data if the current contents of the variable were blank.

This is used in combination with session variables so that whenever possible I'm only going to run the "tests" once which reduces the work required for browsers but unfortunately doesn't have any effect on crawlers since they don't "do" cookies.

If possible I like to avoid using the global.asa on shared hosting because it can be such a pain at times, instead a quick modification to my common include does the job nicely.

My current data source is dynamic - but you can simplify the load process dramatically by using a static data source that's hardcoded into the code and just update it on an adhoc basis.

Also worth noting that it's much easier to write the "you've be banned" page inline rather than using a redirect, 99% of the time the client will only issue the request and not follow it (exploit hunters, formmail spammers etc)

- Tony

[tantalus]

Mmmm. I didn’t expect a reply so soon. Okay here’s where I’m really going to show my ignorance :( I always had a feeling I was going to end up more confused than I was before.

First many thanks for the replies.

Dreamquick when you say ‘by using a static data source that's hardcoded into the code’, are you suggesting that I could for example build an array in the Session_OnStart using…

[*Larbin*]
[*NaverRobot*]
[*www4mail/*]

etc, etc…. and then refer to it using RegExp object as a sub routine say? I’m quite happy to update the global.asa on a adhoc basis, although it would seem easier to upload a text file.

‘write the "you've be banned" page inline’… I’ve never heard of this before, could you expand further?

Lastly, are their problems to having sessions turned on? I don’t understand the concern with this. Is there something I need to know?

Any help much appreciated. This newbie is off to MSDN now… Lesson One!