homepage Welcome to WebmasterWorld Guest from 54.211.47.170
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Is this a valid user agent? and how do I block it?
Is this a valid user agent? and how do I block it?
spiritualseo




msg:4404760
 2:52 pm on Jan 7, 2012 (gmt 0)

Hey guys, is this a valid user agent?

Mozilla/5.0 (X11; U; Linux i686; it-IT; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.25 (jaunty) Firefox/3.8

Have been getting a lot of referral spam mainly from #*$! websites from the above user agent. If this is invalid, any help on how I can block this in htaccess? Thank you!

 

keyplyr




msg:4404814
 7:17 pm on Jan 7, 2012 (gmt 0)

The UA is common. Block by IP address.

dstiles




msg:4404846
 10:41 pm on Jan 7, 2012 (gmt 0)

Jaunty is out of date now - I'm running 10.04 Lucid, which is also being slowly phased out.

I would be wary of the term Ubuntu/9.25. Ubuntu versions are either n.04 or n.10 - Jaunty is 9.04. A search for Ubuntu/9.25 does not produce many results and at least one of those says "forge your UA".

On Lucid I'm currently running firefox 9.1. Ubuntu seemed to be pushing 3.6 until a few months ago - I went straight from 3.6 to 7.something. Can't say I've heard of 3.8 for Ubuntu.

tangor




msg:4404849
 10:56 pm on Jan 7, 2012 (gmt 0)

Sometimes we address the referer, too. Particularly with pron sites... there's aren't that many "words" to deal with. :)

spiritualseo




msg:4404999
 7:48 pm on Jan 8, 2012 (gmt 0)

So would it be safe to block Ubuntu/9.25? Will something like this work:

RewriteCond %{HTTP_USER_AGENT} Ubuntu\ 9.25

dstiles




msg:4405016
 9:26 pm on Jan 8, 2012 (gmt 0)

I would check previous logs and do a bit of research to be sure about blocking it.

As to the rewrite - can't help there, I'm afraid: I don't run htaccess.

lucy24




msg:4405040
 10:36 pm on Jan 8, 2012 (gmt 0)

So would it be safe to block Ubuntu/9.25? Will something like this work:

RewriteCond %{HTTP_USER_AGENT} Ubuntu\ 9.25


You've correctly escaped the literal space, but you also need to escape the literal period. Unless you're expecting visits from "Ubuntu 9425" or "Ubuntu 9a25" or even "Ubuntu 9 25" and want to block those too. Which, come to think of it, wouldn't do any harm. But an unescaped period can lead to unwanted consequences, so it's best to stay in the habit.

And then, ahem, you need a Rule to go with the Cond ;)

spiritualseo




msg:4405271
 5:35 pm on Jan 9, 2012 (gmt 0)

I did check the previous months logs and nearly all traffic from this agent is spam. Successfully blocked this using the following:

RewriteCond %{HTTP_USER_AGENT} Ubuntu\/9\.25 [NC]
RewriteRule .* - [F,L]

keyplyr




msg:4405308
 6:44 pm on Jan 9, 2012 (gmt 0)

Just a FYI

RewriteRule .* - [F,L]


"F" means final and "L" means last, so in effect you are saying the same thing twice. Won't hurt, but a more succinct way of writing this would be:

RewriteRule .* - [F]

lucy24




msg:4405419
 12:25 am on Jan 10, 2012 (gmt 0)

[F] doesn't mean Final, it means Forbidden (think "Fail")-- like Deny from. It's one of a handful of flags that carries an implied [L]. But it's a good habit to include [L] with each separate RewriteRule unless you've got a specific reason to exclude it, so you never leave it out by accident.

In Apache you don't escape / slashes. You're thinking of javascript. Or was that a typo for the same escaped space as before?

keyplyr




msg:4405437
 2:04 am on Jan 10, 2012 (gmt 0)

[F] doesn't mean Final, it means Forbidden
Oh yeah, my bad. Since I only write regex that forbids something I guess I see it as final if there are no more rules. However, I never use both F and L and consider it redundant.

In Apache you don't escape / slashes. You're thinking of javascript. Or was that a typo for the same escaped space as before?

No need to escape forward URL slashes in JS either.

lucy24




msg:4405461
 4:59 am on Jan 10, 2012 (gmt 0)

:) Except in Regular Expressions, because the slashes are what demarcates them.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved