homepage Welcome to WebmasterWorld Guest from 54.226.180.86
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Strange user agent
KevinV




msg:4420797
 9:08 am on Feb 23, 2012 (gmt 0)

Hi all,

New here, but been lurking for years. Thanks to all who post and have pre-answered so many of my questions without me needing to sign up!

I have a user agent claiming to be 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)' hitting one of my sites search page with queries every 5 mins or so, 24/7. Ip's too numerous to mention, but seldom are any *blacklisted* anywhere. It's always searching for people by name, and never seems to go further than the search page (which only lists relevant pages, and doesn't display content).

Does anyone have a clue what this entity might be? I've put a sample of IP's, times and search terms at <snip>

Kev

[edited by: incrediBILL at 10:17 am (utc) on Feb 23, 2012]
[edit reason] NoPersonal URLs Please, include all data in post [/edit]

 

keyplyr




msg:4420818
 10:29 am on Feb 23, 2012 (gmt 0)



Because of the numerous IPs and the frequency, this could be a botnet - many infected machines running a scheduled task.

KevinV




msg:4420820
 10:52 am on Feb 23, 2012 (gmt 0)

Thanks, I did wonder about that. I guess there's little point guessing exactly what it's up to with all those names and variants - last seen searching for ...

Hayes richardson
Hayes richardsson
Hayes richerdson
Hayes richardsan
Hayes richardsen

... from 99.108.182.212 (it uses a different IP for each name and it's variants)

Kev

keyplyr




msg:4420821
 11:04 am on Feb 23, 2012 (gmt 0)

Also, looking through my notes I found that almost all IPs using "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" were bots and a large percentage came from either China, Russia or Asia Pacific. Personally, I block all China ranges on principal :)

wilderness




msg:4420849
 12:34 pm on Feb 23, 2012 (gmt 0)

Also, looking through my notes I found that almost all IPs using "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" were bots and


RewriteCond %{HTTP_USER_AGENT} 5\.1\)$

keyplyr




msg:4421066
 8:39 pm on Feb 23, 2012 (gmt 0)



I block these by other methods.

wilderness




msg:4421078
 9:10 pm on Feb 23, 2012 (gmt 0)

you using a big fly swatter ;)

keyplyr




msg:4421081
 9:23 pm on Feb 23, 2012 (gmt 0)

Who me? What "fly swatter" are you referring to?

wilderness




msg:4421086
 9:35 pm on Feb 23, 2012 (gmt 0)

What "fly swatter" are you referring to?


I block these by other methods.

keyplyr




msg:4421116
 10:33 pm on Feb 23, 2012 (gmt 0)

Sorry wilderness, I can't understand you.

lucy24




msg:4421173
 1:12 am on Feb 24, 2012 (gmt 0)

Sorry wilderness, I can't understand you.


I can. (w himself will testify that this is not always the case.) If you don't block by UA, what do you block by?

fwiw, I recognized that UA instantly. I call it "MSIE generic". It's used by, among other things, all Chinese robots that don't have names-- and a few that do. Kinda doubt any human would use it.

keyplyr




msg:4421182
 1:27 am on Feb 24, 2012 (gmt 0)

If you don't block by UA, what do you block by?

As previously stated, I block China and most of the Asia Pacific by IP, so that covers most all using this UA. The few others I get with that UA are usually already blocked by IP. Occasionally, there are a few *legit* users like library's or military. These I deal with on a case-by-case basis.

A "big fly swatter?" well yeah... China is a big bug!

Seedy




msg:4422587
 11:16 am on Feb 28, 2012 (gmt 0)

RewriteCond %{HTTP_USER_AGENT} 5\.1\)$


Would the above block a user with the below UA as it has "535.11" for it's Applewebkit and Safari version ? Or does the $ state that it must end at that point ?

Mozilla/5.0 (Windows NT 6.0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11

Thank you

wilderness




msg:4422600
 11:59 am on Feb 28, 2012 (gmt 0)

Would the above block a user with the below UA as it has "535.11" for it's Applewebkit and Safari version ? Or does the $ state that it must end at that point ?


the ends with anchor (dollar sign) is the key here to make mod-rewrite only look and compare to that portion of the UA.

In your example it would be
5\.11$
35\.11$
535\.11$

Any of the three would work

\.11$ may also work, however over time, I've learned to be cautious about opening a line with an escape.

Seedy




msg:4422833
 9:13 pm on Feb 28, 2012 (gmt 0)

Thank you wilderness. I was actually trying to be cautious. My poorly written question was meant to ask if by using the quoted rule I may accidentally block those other UA's but I can see from you reply that it would not.
Many thanks for your reply.

dstiles




msg:4422859
 10:21 pm on Feb 28, 2012 (gmt 0)

Of course, that block will only work with this week's chrome browser. Next week the version numbers are likely to change (last week-ish it ended in 7).

And, of course, this will kill all versions of safari of that version number, chrome or not, unless tied to a specific IP or other header fields.

tangor




msg:4422902
 12:30 am on Feb 29, 2012 (gmt 0)

Keep it a bit more simple: MSIE 6\.0

After all, who's running that these days? Go for the nibble, not the over-large sledgehammer. .htaccess 6 and all those others lesser in value, too. These days I'm loving my 403's... and that tiny tiny (teeny tiny) 403 served and homemade script that strips those out for analysis on what remains.

motorhaven




msg:4423098
 2:14 pm on Feb 29, 2012 (gmt 0)

Just a heads up, if you're running NetDNA or similar content delivery network their pull cache uses this MSIE 6.0 user agent.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved