Forum Moderators: open

Message Too Old, No Replies

AOL users running bots.

any tips for halting them?

         

davelms

8:20 pm on Dec 12, 2005 (gmt 0)

10+ Year Member



In the past few weeks I've been targetted by some AOL US users - maybe user singular - downloading my entire site, not taking images, just pages and tripping my "bad bot" traps. The user-agent is always:

Mozilla/4.0 (compatible ; MSIE 6.0; Windows NT 5.1)

(thats a cut and paste)

IP ranges, here's some examples:

172.213.159.**
172.201.131.***
172.188.155.***
172.202.5.***
172.212.129.***
172.216.177.***

Now, having experienced Firefox's FasterFox extension recently, and the browser accidentally tripping the bad bot trap I wondered:

a) has AOL implemented something similar, US only? And if so how to detect?
b) or if this is a bot with a rogue user-agent? And if so what would people recommend to catch it from the details above.
c) anything else I might have missed?

[edited by: volatilegx at 3:00 pm (utc) on Dec. 13, 2005]
[edit reason] obscured IP addresses [/edit]

Pfui

3:15 am on Dec 13, 2005 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



A few thoughts...

1.) If an AOLer has their own ISP account they can subscribe to AOL's 'bring your own access' type of membership (unlimited use for less $/month).

Because they can run any browser through their AOL connection and the IP address will be one of AOL's, there's a good chance that your robotic hits come from someone actually running a bot.

2.) The UA is suspect because of the space *before* the semi-colon. If someone has doctored the UA string (doable in any number of bots, browsers and FF extensions), then chances increase that the accesses coming from -- or perhaps more correctly, running through -- AOL are bots.

At least with that atypical string, you can mod_rewrite the UA into oblivion -- until someone catches on and changes it.

3.) With strings almost commonly (let alone maliciously) malleable nowadays, and rogue and/or robots.txt-abusing bots increasingly rampant, I'm looking into installing some sort of bandwidth throttle module, m'self. If you're an Apache user, you might want to, too.

davelms

7:34 am on Dec 13, 2005 (gmt 0)

10+ Year Member



Yes, just checked through all my logs, these rogue hits do have the space before the ; in the UA. So yes, I can target that...

thankyou.