UA blocking of 80legs bot?

using .htaccess to block 80legs bot

[classifieds]

I'm trying to block the 80legs bot on the UA and I can't get it to match. Suggestions or advice would be appreciated.

RewriteCond %{HTTP_USER_AGENT} 80legs [NC,OR]
RewriteRule . - [F,L]

Here's an entry from my httpd logfile:

173.50.159.nnn- - [26/May/2010:05:54:40 -0400] "GET /list_of_Mardagroup.html HTTP/1.1" 200 3359 "-" "Mozilla/5.0 (compatible; 008/0.83; http://www.80legs.com/spider.html;) Gecko/2008032620"

[edited by: incrediBILL at 10:24 am (utc) on May 26, 2010]
[edit reason] Obscured IPs, fixed formatting [/edit]

[jdMorgan]

Remove the [OR] from the RewriteCond, if that is your only RewriteCond.

The last RewriteCond in a 'list' of RewriteConds should never have an [OR] flag, as it makes no logical sense.

The [L] flag is also not needed: [F] implies [L], so [L] is redundant.

[added]
Note also that by using "." as the RewriteRule pattern, your rule *will not* block access to your "home page" if this code is located in .htaccess. I'd suggest:

RewriteCond %{HTTP_USER_AGENT} 80legs [NC]
RewriteRule ^ - [F] 

Generally, the only resources you want to exclude from 403 access control rules are your robots.txt file and your custom 403 error document (if you have one).
[/added]

Jim

[maximillianos]

For some reason we had to do the following to get it to work:

RewriteCond %{HTTP_USER_AGENT} ^.*80legs.* [NC]

Without the .* on each side it was not matching for us?

Anyway, just thought I would add my experience.

[nevermind - I was using the ^ symbol, which changed the logic! - works fine without my mod]

[dstiles]

Much as I dislike distributed bots, 80legs (and the REAL MJ12) respects robots.txt. Since adding a disallow I haven't seen them.

[Pfui]

Their history is lousy -- [webmasterworld.com...] -- and they swarm. I chop 'em off at their 80knees.

[keyplyr]

Much as I dislike distributed bots, 80legs (and the REAL MJ12) respects robots.txt. Since adding a disallow I haven't seen them. - dstiles

I agree:

User-agent: 008
Disallow: / 

[Dijkgraaf]

@Pfui How have they been recently though?
I don't get a lot of hits from that bot.
It still has the habit of requesting robots.txt before most requests but that may be due to the time between the requests. I did request one page twice in one minute.
Apart from that I've had no problems with them recently (last three months).

[MxAngel]

First time I got hit by them:

Period: 26/Nov/2010:15:58:15 -0700 - 26/Nov/2010:19:53:25 -0700
Search "80legs" (2409 hits in 1 files)

1 request for robots.txt from 64.125.222.16.available.above.net, followed by c-68-44-182-107.hsd1.pa.comcast.net requesting pages, and then a huge amount of requests from different IP's / ISP ranges.

They're out with me.