block request by keyword

[lws123]

Hello all,

I am new here and new to Apache but wondered if someone might help me with a mod_rewrite issue I have.

The problem I have is that search engines keep requesting certain urls every few seconds which triggers resource heavy sql queries (a custom mod for shopping cart software). These queries keep bringing the server to a halt.

So at the moment I am banning googlebot and microsoft ips so that server stays up which I dont want to do for long.

I need to give a 403 to certain IPs when they request a URL that contains the keyword "xsearch" anywhere in the URL.

I've tried what feels like a million things to make this happen but they don't work.

The closest I have gotten is as follows which is for a chinese bot.

RewriteCond %{REMOTE_ADDR} ^180\.76\.5\..*
RewriteCond %{HTTP_REFERER} ^(http://)?(www\.)?.*(-|.)xsearch(-|.).*$ [NC]
RewriteRule .* - [F,L]

This works if the previous page looked at contained 'xsearch' but when I try and use this to create a {REQUEST_URI} condition instead of a {HTTP_REFERER} it doesn't work.

I only want to stop bots from firing any URLs that contain "xsearch" anywhere in the URL but let them do whatever they want with any other URL that doesnt contain that phrase.

Would very much appreciate any help, I have basically no idea what I'm doing with mod_rewrite - my efforts have been purely based on googling!

Thank you!

[wilderness]

do a search (lower portion of page)

request_URI

[lws123]

Hi,

Thanks for the reply. I've looked through and can't find anything other than something I have tried to adapt to my needs :

RewriteRule ^xsearch/([^/]+)/([^-]+)-([^.]+)\.html/?$ - [F,L]

It doesnt seem to do anything, and from my limited knowledge of syntaxes it seems to me that the above would only look for 'xsearch' the beginning of url and not anywhere in URL anyway.

Any further help appreciated

[g1smd]

The most basic form blocks requests with xsearch anywhere in the path:

RewriteRule xsearch - [F]

Slightly more complex, blocking any request with xsearch anywhere in the query string:

RewriteCond %{QUERY_STRING} xsearch
RewriteRule . - [F]

The above rules block these requests for all users. Add more RewriteConds to be more selective in which users the rules apply to.

You do not need to be looking at referrer data at all, and you should never rely on that being present or, when present, even being true.

[lws123]

Thanks so much g1smd, the following didn't want to work:

RewriteRule xsearch - [F]

However the second you provided works perfectly:

RewriteCond %{QUERY_STRING} xsearch
RewriteRule . - [F]


The solution is so simple compared to all the very confusing things I've tried. Yes I did spend hours on this!

Thanks again.
-Very happy new forum user!

[g1smd]

The RegEx pattern in the rule is the first bit to be evaluated.

If it is a match, processing then looks at the RewriteCond(s) if present.

Finally the rule target is processed.

[lws123]

Ok ty :)