homepage Welcome to WebmasterWorld Guest from 54.198.94.76
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
User agent chomping bandwidth
Syntax error?
didibreakit




msg:4329921
 1:45 pm on Jun 23, 2011 (gmt 0)

Hello everyone,

I am trying to block a user agent that keeps hitting my site, evidently from behind proxies, and chomping away at bandwidth.

Can anyone point out what I'm doing wrong?

Thanks



I would like to block:

Mozilla/5.0\ (Windows NT 5.1) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.100 Safari/534.30

I thought this would work but no joy:

RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5.0\ \(Windows\ NT\ 5.1\)\ AppleWebKit/534.30\ (KHTML,\ like\ Gecko\)\ Chrome/12.0.742.100\ Safari/534.30$
RewriteRule ^.*$ deny.html [L]

 

thetrasher




msg:4330102
 6:50 pm on Jun 23, 2011 (gmt 0)

(KHTML -> \(KHTML
. -> \.



Mozilla/5.0\ (Windows NT 5.1) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.100 Safari/534.30

There is a backslash?!

Pfui




msg:4330171
 8:21 pm on Jun 23, 2011 (gmt 0)

Presuming the backslash in didibreakit's to-block string is a tpyo...

I recommend blocking the IPs/Hosts/proxies and not the UA. My log on a small site shows 323 hits using that UA thus far this month - a not insignificant number. In just the last hour, visitors using that UA hailed from broadviewnet.net to verizon.net.

If you still want to slam the door, here's the full string showing the missing backslashes mentioned by thethrasher. You also need to backslash commas:

Mozilla/5\.0\ \(Windows\ NT\ 5\.1\)\ AppleWebKit/534\.30\ \(KHTML\,\ like\ Gecko\)\ Chrome/12\.0\.742\.100\ Safari/534\.30

(Aside: Here's hoping I didn't typo!)

But that's a crazily detailed string. A shorter way might be to block on the version:

Mozilla.*Chrome/12\.0\.742\.100

Just my 2-bytes.

P.S.

If the backslash is part of the to-block string, welllll, I dunno. That's one for Apache Forum mod jdMorgan. Unless putting the backslash in an array of not-okay chars might work to nullify its escape effect? Or not... This is UNtested (& veering into Apache territory):

RewriteCond %{HTTP_USER_AGENT} (\+|\|\*)
RewriteRule .* - [F,L]

g1smd




msg:4330176
 8:26 pm on Jun 23, 2011 (gmt 0)

To escape a literal backslash add a backslash: \\ but this is unlikely to be found in a User Agent.

[F] implies [L]. When [F] is used, [L] is not required. Use [F] only.

I see that User Agent and several similar User Agents coming from Google IP ranges in recent days.

In those cases, both images and stylesheets have also been pulled and I thought it was a real browser.

lucy24




msg:4330189
 8:50 pm on Jun 23, 2011 (gmt 0)

There is a backslash?!

I hope so, because the exact string

Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.100 Safari/534.30

is a real UA used by humans. The 12.0.742.100 must be a new release of Chrome, because it first showed up in my logs on the 15th (yes, I checked). Various IPs and various referers of unquestioned, er, human-ness.

Edit: Oops, typed too slow, since g1 above said the exact same thing I said. (If he had said something different, he would be right and I would be wrong.)

g1smd




msg:4330196
 9:00 pm on Jun 23, 2011 (gmt 0)

I've seen a number of similar User Agents in recent days. I've assumed they are human.

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.91 Safari/534.30"

"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_5) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.100 Safari/534.30"

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.24 Safari/535.1"

"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1"
didibreakit




msg:4331275
 8:55 pm on Jun 26, 2011 (gmt 0)

Chrome 12? Wow. Who knew? Well obviously you kind folks did. Thank you for your help.

g1smd




msg:4331292
 9:30 pm on Jun 26, 2011 (gmt 0)

12 is the current stable version. 13 and 14 are beta versions.

It is fitting that Google employees are using bleeding edge releases.

[en.wikipedia.org...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved