Banning by user agent

[brokaddr]

I have a problematic host that is consistently trying to scrape my site.

Which is the most effective method of blocking, in relation to bandwidth/server consumption if the pest is persistent?

mod_security?

SecRule REQUEST_HEADERS:REMOTE_HOST "host-name-here" deny,status:403

- this doesn't seem to work.

htaccess?

 SetEnvIfNoCase Remote_Host "host-name-here" bad_bot

[wilderness]

This old thread is enough to get you started.

Close to Perfect htaccess [webmasterworld.com]

[incrediBILL]

I'm not sure of which is the most efficient but considering mod_security is yet another layer of add-ons, I'd assume it would be slightly less efficient just having it there in the first place.

Overall REMOTE_HOST is grossly inefficient as it forces a reverse DNS lookup which can burn a lot of time opposed to simply blocking by IP range.

I don't mind reverse DNS look ups if they're cached, which I do in my code but I can't voich for REMOTE_HOST, so it doesn't repetitively do it every time the server encounters the same IP in a short time period.

This old thread is enough to get you started.

Yeah, but that's blacklisting and the number of user agents to block is in the thousands now and the amount of linear processing added to every Apache process is ridiculously inefficient.

Do the same basic code but whitelisting and it's short, efficient and permanently effective, you're good to go.

[wilderness]

I have a problematic host

Looks singular to me ;)

[brokaddr]

Overall REMOTE_HOST is grossly inefficient as it forces a reverse DNS lookup which can burn a lot of time opposed to simply blocking by IP range.

Good to know!

I've already got a whitelist reverse DNS checker up for the primary search engines, so if I rewrote that to boot the pest(s) I should be OK.

Looks singular to me ;)

1 really bad one, that's almost un-bannable at this point; every method I've tried thus far seems futile. As soon as I block a range, it comes back with another range... always has the same hostname, though.

[wilderness]

1 really bad one, that's almost un-bannable at this point; every method I've tried thus far seems futile. As soon as I block a range, it comes back with another range... always has the same hostname, though.

Than your NOT defining the range far enough.

[incrediBILL]

it comes back with another range... always has the same hostname

I have a few I had to block by host name for the same reason.

Sometimes there just is no choice.

[lucy24]

mod_security?
SecRule REQUEST_HEADERS:REMOTE_HOST "host-name-here" deny,status:403 - this doesn't seem to work.

htaccess?
SetEnvIfNoCase Remote_Host "host-name-here" bad_bot

By yawn-provoking coincidence I have just this minute come from an unrelated forum where someone had exactly the same kind of "Is it hotter in New York than in the summer?" question.*

You mean mod_security or mod_setenvif. Why bother with the host name at all? Somewhere behind the name is an IP address-- and it's less likely to be faked than anything else you could block. If you're in doubt about the full range, just make it bigger. If for example it claims to be
aa.bb.cc.0/19
but your raw logs don't turn up anything from the rest of aa.bb., just block the whole /16. Or /15 or /14 if you haven't met any humans from there either.

I don't know whether anyone has done a rigorous speed comparison on mod_rewrite using RegExes vs mod_auth-thingie using CIDR ranges. (This is assuming you're not lucky enough to be in Apache 2.4 yet.) Gut feeling says that anything in CIDR form will be faster. But unless you've got an absolutely enormous site-- which you've said you don't-- the difference isn't likely to be significant.


* Answer: Yes.

[wilderness]

Why bother with the host name at all? Somewhere behind the name is an IP address-- and it's less likely to be faked than anything else you could block. If you're in doubt about the full range, just make it bigger. If for example it claims to be
aa.bb.cc.0/19
but your raw logs don't turn up anything from the rest of aa.bb., just block the whole /16. Or /15 or /14 if you haven't met any humans from there either.

I agree, and furthermore, if your unable to focus on the IP range?
Simply start denying Class A's (/8) temporarily, and then expanding them in follow up.

[MickeyRoush]

Why bother with the host name at all?

Wouldn't this be beneficial if the host adds IP ranges to their existing ones? That way, you would be covered for future events.

[lucy24]

But only at the expense of speed loss everywhere. Especially if you're using an exclusion method that looks at all requests all the time. Since each request is an island, your htaccess doesn't know that those eighteen consecutive image requests come from a page that the visitor has already got permission to visit, so there's rarely any point to checking permissions* all over again. (I'm talking here about generic allows and denies, not the separate issue of hotlinks.)

Hm. Would it save any time overall if the WHOLE list of Deny from... directives were placed inside a Files envelope that constrained it all to .html requests? Or would the envelope itself add even more time to request processing? No, I have no idea why this question never occurred to me before.


* "Permission" in the casual sense, not the 401 sense.