User Agent "Firefox"

[marodhum]

I got hits from this bot/crawler with user agent "firefox" from 24.185.86.nnn

24.185.86.nnn - "GET /my_page.html HTTP/1.0" 200 7380 "-" "Firefox"
24.185.86.nnn - "GET /my_page1.html HTTP/1.0" 200 6894 "-" "Firefox"
24.185.86.nnn - "GET /my_page2.html HTTP/1.0" 200 10461 "-" "Firefox"

Since it is asking only html files, so my guess that it must be a bot.
Can i ban it by mod rewrite? Like this,
RewriteCond %{HTTP_user_agent} ^Firefox$ [NC]
RewriteRule ^.*$ X.html [F]

[edited by: encyclo at 4:07 pm (utc) on Mar. 8, 2008]
[edit reason] fixed formatting [/edit]

[Staffa]

I saw the same UA today coming from 24.186.212.nnn which is another range of the same Cablevision supplier.
I'll see if it comes back, then it gets banned.

[wilderness]

Can i ban it by mod rewrite? Like this,
RewriteCond %{HTTP_user_agent} ^Firefox$ [NC]
RewriteRule ^.*$ X.html [F]

I would suggest changing your second line to the following:

RewriteRule .* - [F]

Firefox always provides a more extensive and standard UA, which includes a version.

Here's two examples:
"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.12) Gecko/20080201 Firefox/2.0.0.12"

"Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.12) Gecko/20070508 Firefox/1.5.0.12"

As a result, it's safe to assume that your visitor is using a fake UA.
Personally, I'd include the IP range as well.

Don

[marodhum]

Thanks everybody for the input. I have already banned the ip range, but also wants to ban the fake UA, in case somebody else going to use it.

I would suggest changing your second line to the following:

RewriteRule .* - [F]

Don, i will be obliged, if you please explain the difference between the two rules.

[wilderness]

if you please explain the difference between the two rules.

marodhum,

Perhaps keyplr, Jim or amother may provide your explaination?

Not sure what your use of the dollar sign adds or even the leading caret.

Further, your rewriting the denial to an "x.html" page, when the most effective practice for these fake bots and other harvest attempts is a straight-forward denial (which I've provided).
Why provide them the fake bot or harvester with more ammunition, after all, they are in fact being less than forthright in their visits and/or protocols when visiting your website (s).

Don

[blend27]

A lot of them(fake bots and other harvest attempts) do not contain ")" bracket. I was looking for a simple rule (regexp) here [webmasterworld.com...] , that would allow me to block/filter a lot of junk.

In other words if the user agent does not contain ")" or a "(" it cuts down on the list of rules that that need to be processed. There are some UAs for Nokia Cell phone browsers that do not contain the curly though.

Blend27

[keyplyr]

Not sure what your use of the dollar sign adds or even the leading caret - Don

^ is used to signify "starts with"

$ is used to signify "ends with"

Using these when possible helps efficiency and may save processing resources and time.

In the last line, Marodhum is sending this UA to X.html but it's messy. I would suggest cleaning that code up and doing it as Don suggests:

RewriteRule .* - [F]

Then forwarding like this to give the correct status code.

ErrorDocument 403 /X.html

(however, I suggest using all lower-case characters in URLs)

[incrediBILL]

I've seen just "FIREFOX" from the following IPs:

170.140.150.x
150.101.241.x
125.33.46.x
69.113.83.x
24.186.215.x
24.186.212.x
24.186.165.x
24.185.86.x
24.185.86.x
24.184.102.x

Looks like DSL and cable lines, no clue what they're up to.