homepage Welcome to WebmasterWorld Guest from 54.211.219.178
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
WebSense Hiding Behind Rotating User Agents
Internet Security Company Tricks Site Security
Not academic




msg:3692077
 11:59 am on Jul 7, 2008 (gmt 0)

Hello People,

I have looked through the board descriptions and can not find one that exactly fits....so here I am... Please excuse if I am in the wrong board.

My very small site gets a visit from an ip static-208-80-193-**.as13448.com perhaps 10 times a day.

In the past this site and other address's in the same block would read my robots.txt and then completly ignore it....so the entire range now gets a 403, just for the hell of it.

Until today the User Agent has always been plausable (at least to me it is) The UA is always different on each visit as is the precise IP address....always from within that address range thopugh.

Today it started showing this as the agent...

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Sky Broadband; Sky Broadband)

and variations of it except for the Sky Broadband part

I am now fairly certain that this is one of those rotaing User Agent strings I have heard of. I'm a Sky Broadband user myself and the above string is not how it "used to" appear from my browser.

Can anyone offer some thoughts on this? Am I right or off beam?

Many Thanks in advance

[edited by: jatar_k at 4:13 pm (utc) on July 7, 2008]

[edited by: incrediBILL at 10:34 pm (utc) on July 10, 2008]
[edit reason] Obscured IPs [/edit]

 

wilderness




msg:3692672
 9:44 pm on Jul 7, 2008 (gmt 0)

Perhaps this search [google.com] will help

In addition the change in UA may have some reflection on either the version and/or updates of "Sky" software?

wilderness




msg:3692763
 11:55 pm on Jul 7, 2008 (gmt 0)

The following from May with an IP range of a content filter:

208.80.193.zz - - [07/May/2008:00:26:40 -0500] "GET / HTTP/1.1" 200 6774 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Sky Broadband; FunWebProducts; Sky Broadband; Sky Broadband)"

Thus far during July and on one of my sites, I've the following UA's which include "Sky Brodband":

"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Sky Broadband; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"

"Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.0; Windows NT 5.1; Sky Broadband; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0)"

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0)"

"Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Sky Broadband; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"

There were ALL referred by on-topic searches.
Three were from different-RIPE IP ranges (Class A or B) and denied access to my sites.
The shortest UA was an ARIN range and allowed access.

As a result, I may only assume that Sky is not using proxies. At least consistently.

Not academic




msg:3692930
 5:44 am on Jul 8, 2008 (gmt 0)

This address range has nothing to do with Sky as far as I know.

Sky Broadband is as far as I know a UK only ISP. Am I correct with that?

BUT in any case, as I said earlier the UA changes on every visit and the IP changes. To elaborate on that further, One moment it might say its a Mac the next its running IE 6 or 7 and the IP might change by one digit in the final block of four. Next time the exact same IP is used its a completly different machine....sometimes within just a minute or so of the previous visit.

<snip> comment removed in thread cleanup</snip>

Comment
=======

Exactly why my message has been moved to this forum is beyond me. I do not believe my enquiry has anything to do with a search engine, nor do I suggest it. Its my personal view that the whole issue is actually a range of IP address's that are being used for an unauthorised purpose by for want of any other name a hacker.

[edited by: Not_academic at 5:59 am (utc) on July 8, 2008]

[edited by: engine at 11:54 am (utc) on July 8, 2008]

[edited by: incrediBILL at 4:22 am (utc) on July 11, 2008]
[edit reason] thread cleanup [/edit]

incrediBILL




msg:3692938
 6:20 am on Jul 8, 2008 (gmt 0)

Exactly why my message has been moved to this forum is beyond me. I do not believe my enquiry has anything to do with a search engine, nor do I suggest it. Its my personal view that the whole issue is actually a range of IP address's that are being used for an unauthorised purpose by for want of any other name a hacker.

Not_academic, being new to WebmasterWorld let me explain.

Don't let the title of the forum fool you, we're the IP address experts, we know IPs best, that's why it was moved in here so you got some of the best minds on the subject that you see posting in this forum all the time.

<snip> comment removed in thread cleanup</snip>

[edited by: incrediBILL at 4:23 am (utc) on July 11, 2008]

thetrasher




msg:3693125
 11:58 am on Jul 8, 2008 (gmt 0)

My very small site gets a visit from an ip either the same as or very close to static-208-80-193-**.as13448.com perhaps 10 times a day

I see 208.80.193/24 coming to my homepage with different fake UAs, too. Since 2007-12-01 with rising frequency, now once day. I don't know why Websense is interested in my root page and why they (try to) pretend to use infected browsers.

deny from 208.80.192.0/21

"Websense Stealth Crawler Bypassing Security?"

Samizdata




msg:3693141
 12:15 pm on Jul 8, 2008 (gmt 0)

Thanks for clarifying that it was Websense rather than the other nuisances I mentioned.

There are plenty of posts on WebmasterWorld about them if you search on the name.

I don't know why Websense is interested in my root page

They believe it is their right to robotically inspect your site and all others.

...

Not academic




msg:3693656
 8:19 pm on Jul 8, 2008 (gmt 0)

Hello again, Well I am not sure it is Websense (in my case that is). Remembering that my site really is very very small and does not get many new users (Its strongly a family site).......

I am now getting visits from Websense immediately followed by calls that are identical in all respects except apparent origin, only moments after Websense, except they are now coming from Global Crossing (Scotsdale) AND Wanadoo based in Utrecht, Netherlands. The UA's are as good as identical.

So perhaps I am getting rotating UA's in conjunction with spoofed IP address's from (for want of a less polite description, a hacker of some sort)

or, perhaps someone is having a little fun with me......

<snip> comment removed in thread cleanup</snip>

[edited by: Not_academic at 8:27 pm (utc) on July 8, 2008]

[edited by: incrediBILL at 4:22 am (utc) on July 11, 2008]
[edit reason] thread cleanup [/edit]

incrediBILL




msg:3693930
 1:30 am on Jul 9, 2008 (gmt 0)

Well I am not sure it is Websense (in my case that is)

It is WebSense in your case as the whois for the IP and the domain referenced in the reverse DNS both resolve to "Websense, Inc.".

Hope that clears it up.

[edited by: incrediBILL at 10:45 pm (utc) on July 10, 2008]

Megaclinium




msg:3694099
 6:28 am on Jul 9, 2008 (gmt 0)

I get hits from this same guy 208.80.193.xx with rotating UA also.
Just recently it comes with a huge long hex registry key type value at end. I'm masking it out in case they're trying to connect forum posters here with websites they hit.

they never get the images from my main page and never go after anything else. I haven't banned them yet because of that. Maybe they're testing root web page for infective agents?

"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; {A1B2C3D4-E5F6-G7H8-I9J0-K1L2M3N4O5}; SV1; .NET CLR 1.1.4322)"

incrediBILL




msg:3694125
 7:11 am on Jul 9, 2008 (gmt 0)

Sorry, but it's still websense if you do a WHOIS on the IP

wilderness




msg:3695778
 9:45 pm on Jul 10, 2008 (gmt 0)


System: The following message was cut out of thread at: http://www.webmasterworld.com/search_engine_spiders/3695122.htm [webmasterworld.com] by incredibill - 2:16 pm on July 10, 2008 <small>(PST -8)</small>


Yankee Doodle ;)

208.80.193.* - - [10/Jul/2008:22:06:01 +0100] "GET / HTTP/1.0" 301 391 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; {C13455BA-974A-C95C-848E-98CF3CA8035D}; .NET CLR 1.1.4322; .NET CLR 1.0.3705)"
208.80.193.* - - [10/Jul/2008:22:06:01 +0100] "GET / HTTP/1.0" 200 8040 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Wanadoo042NLPP01HCFFFFFFE8; SV1; SIMBAR={2B30982A-505F-4a6c-8941-2F678BA39B6B})"

I've pretty much left websense alone. They generally visit multiple times each day and only grab the sites main pages.

This the last draw.

deny from 208.80.192.0/21

[edited by: incrediBILL at 10:18 pm (utc) on July 10, 2008]
[edit reason] Obscured IPs [/edit]

incrediBILL




msg:3695803
 10:26 pm on Jul 10, 2008 (gmt 0)

They've only attempted to access 1700+ pages from my site this year which is reasonable for an internet security company in 6 months.

However, it's the deceptive way they attempt to do it which is why I've had them blocked for a long time with the rotative user agent.

The part that amuses me is they insist on using the "Konqueror" browser UA which is so obscure in the first place that it put a big neon sign out saying "LOOK AT ME! SOMETHING FISHY IS GOING ON! CHECK MY IP ADDRESS!"

wilderness




msg:3695814
 10:34 pm on Jul 10, 2008 (gmt 0)

Bill,
I may eventually add all their (Websense) IP ranges to my denies. I did accumualte five, although I seem to recall that they also utilize an odd-sub-net range unnder PSI or another, which I was unable to locate.

wilderness




msg:3695824
 10:37 pm on Jul 10, 2008 (gmt 0)

BTW, it was actually the Wannadoo reference in the Websense UA that prompted my "last draw", rather than the websense Ip ;)

Don

idiotgirl




msg:3695841
 11:05 pm on Jul 10, 2008 (gmt 0)

While we're on the subject of Websense, I don't recall seeing this UA from them until today:

208.80.193.XX - - [11/Jul/2008:01:27:19 -0400] "GET / HTTP/1.0" 200 3010 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; CashRegister3; CashRegister1)"

Is this some kind of security pre-checker for e-commerce sites for people submitting orders from their employer's computers? It only hit the main page, and not even a shopping page. Kinda weird.

Samizdata




msg:3695887
 12:31 am on Jul 11, 2008 (gmt 0)

Websense ThreatSeeker Network scans over 595 million websites per week, searching for threats.

They only try to take the home page (no scripts or other files) and use many user-agents.

They attempt to access my sites (and yours) by deception in order to make their profits.

Websense Web Security Suite is a leading web security solution that protects organizations from known and new web-based threats.

Their methods are both dishonest and (in my case) hopelessly ineffective. But...

"No one is this world, so far as I know - and I have researched the records for years, and employed agents to help me - has ever lost money by underestimating the intelligence of the great masses of the plain people." (H L Mencken)

...

Romeo




msg:3696246
 12:18 pm on Jul 11, 2008 (gmt 0)

The part that amuses me is they insist on using the "Konqueror" browser UA which is so obscure in the first place that it put a big neon sign out saying "LOOK AT ME! SOMETHING FISHY IS GOING ON! CHECK MY IP ADDRESS!"

ahem, hmm, well ... why do you think that?

'Konqueror' is the default browser on a Linux KDE desktop, and since it is smaller and faster than Firefox, I am using it all the time for my private web surfing ...
Didn't know I am considered obscure due to this ... obscure, yes, but just due to this?

And in case the Konqueror may have difficulties in rendering some pages, I also have the Gnome 'Epiphany' browser on my Kubuntu desktop. Another obscure one, yes, I already know.

From the large variety of UAs mentioned in this thread, I would assume that they at Websense may just randomly pick them from their own web server log files to insert into their bot ...

Who are the users of Websense?
Would blocking their bot be like shooting in the own feet because our sites may get flagged and their customer base may be driven away?

Perhaps we should give them a blank 200 page or a 'Lore ipsum' page instead of a harsh 403.

Kind regards,
R.

blend27




msg:3696335
 2:40 pm on Jul 11, 2008 (gmt 0)

--- Who are the users of Websense? ---

A lot of Government agencies sit behind Websense as a content filtering tool in USA.
Lots of sites are getting filtered, depends on policies set by the admin. So in the sense it does increase productivity.

I would not recommend blocking it.

incrediBILL




msg:3696434
 4:11 pm on Jul 11, 2008 (gmt 0)

Didn't know I am considered obscure due to this ... obscure, yes, but just due to this?

Being obscure doesn't mean it's bad, it means it's only a very small percentage of people are using it compared to MSIE, Firefox, Opera, etc.

When you see one source that skews the statistics for "Konqueror", in this case WebSense, it sets off a big red flag because it's a single source for a significant amount of traffic using a relatively unseen user agent.

I would not recommend blocking it.

Whoops! Too late...

They stand out like a sore thumb, easily spotted, easily spoofed.

They've been getting a small response page to keep the bandwidth down much like others did to AVG for quite some time now.

[edited by: incrediBILL at 4:12 pm (utc) on July 11, 2008]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved