homepage Welcome to WebmasterWorld Guest from 54.227.89.236
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
verify-Compliance Notice
An unusual referer?
dstiles




msg:3859465
 8:31 pm on Feb 27, 2009 (gmt 0)

I've been getting a few referers recently with this, along the lines of:

http://www.example.com/80:verify-Compliance_Notice/garbagecodestring

Obviously I don't have such a URL so it's falling into the 404 handler and incidentally blocking the user by IP.

They seem to be unique accesses, although tracing the IP for the latest one turned up two rejected accesses to a different domain than the original access (but similar town-related content).

All I can find about it on google seems to point to it being some phone unlocker, which I find hard to credit in the circumstances unless it's a wandering bot looking for sites that offer phone unlockers. But why then go on to access another domain on the same server with what looks like a genuine request?

Any ideas, guys?

 

wilderness




msg:3859538
 10:38 pm on Feb 27, 2009 (gmt 0)

I've a reference to something very similar from July of last year.

The refer was an on-topic search through MSn search, however the requested page was began with /notified-Compliance_Notify, which resulted in a 404.

After two requests of this unusual pattern, the visitor resumed normal activity. IF your able to call requesting the same page six times in succession within the same second ;)

The IP range was a bank, which I assumed to be some kind of open proxy.

I denied the range.

dstiles




msg:3859601
 12:45 am on Feb 28, 2009 (gmt 0)

Sorry, got the port identifier syntax wrong in the first posting.

By checking for "verif" for this month I've now found others in the trap logs, all to domains with town/county-related content...

:80/verify-SNL_Splash?garbage
203.4.237.nnn Australia ?broadband

:80/verify-NotifyUser2?garbage
163.166.8.nnn British travel company

:80/verify-Compliance_Notice?garbage
80.249.57.nnn local UK council (from two different IPs in 18 seconds)

All used variations on the "standard" MSIE XP UA.

Referers were yahoo uk, yahoo au, google uk and one of my other sites (so it must have been a good hit on that one or it would have been trapped).

This is beginning to look like some kind of site verification. No sign of a proxy being involved but it could have been completely stealthed, of course.

Mokita




msg:3859690
 5:21 am on Feb 28, 2009 (gmt 0)

dstiles wrote (in part):
:80/verify-SNL_Splash?garbage
203.4.237.nnn Australia ?broadband

That range is not just any broadband, it belongs to Amalgamated Television Services (ATN Channel 7), one of Australia's biggest television broadcasting companies.

I've been getting the same odd requests from that IP range for years. But my investigations have yielded nothing to tell me what causes them. I have surmised that the company has installed some sort of a weird browser add-on on all their computers, but have no evidence for that. I have seen the same sort of requests coming from other IPs occasionally too, but haven't bothered investigating as I don't block them.

I have not blocked them, because ATN 7 are regular clients of the company whose site I monitor. And after the initial 404'd requests, they resume normal transmission (er, I meant, they assume normal human browsing behaviour). ;)

The requests always come in a pair like this:
203.4.237.nnn - - [25/Jan/2009:14:39:19 +1000] "GET <base-64 encoded url> HTTP/1.1" 404 4297 "http://www.google.com.au/search?hl=en&cr=countryAU&sa=X&oi=spell&resnum=0&ct=result&cd=1&q=widget&spell=1" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322)"

203.4.237.nnn - - [25/Jan/2009:14:39:19 +1000] "GET <base-64 encoded url> HTTP/1.1" 404 4295 "http://www.google.com.au/search?hl=en&cr=countryAU&sa=X&oi=spell&resnum=0&ct=result&cd=1&q=widget&spell=1" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 1.1.4322)"

I'd certainly like to know what purpose they are supposed to have, because it seems guaranteed that those requests will get 404s on any site they visit, unless it is intended for their own intranet.

[edited by: tedster at 11:17 am (utc) on Feb. 28, 2009]
[edit reason] obscure the encoded url [/edit]

phred




msg:3859732
 10:03 am on Feb 28, 2009 (gmt 0)

This guy may have the answer. Seems a security product called ProxySG uses requests like verify-SNL_Splash. Meant to be on the LAN side but apparently it may be possible to attack the product and the external hits you are seeing may be bot's probing to find users of ProxySG.

Google "ProxySG hypothesis"

Cheers,
Phred

Mokita




msg:3859737
 10:35 am on Feb 28, 2009 (gmt 0)

phred wrote:
Meant to be on the LAN side

I fully accept that part of the hypothesis - I mentioned intranet in my last post.

... but apparently it may be possible to attack the product and the external hits you are seeing may be bot's probing to find users of ProxySG.

If that is true in our case, it is the most advanced bot you could possibly dream up. It not only correctly fills in and submits order forms, but it also responds intelligently from the email address it provided (belonging to ATN 7). And then on the appointed day, the "bot" (masquerading in human form) either collects the product from the office or receives and signs for it by courier delivery and pays for it all in advance via the company credit card. And it does this multiple times per year.

Just maybe, I don't quite believe this is bot behaviour. ;)

[edited by: Mokita at 10:36 am (utc) on Feb. 28, 2009]

dstiles




msg:3860015
 9:28 pm on Feb 28, 2009 (gmt 0)

Ah. So Blue Coat rears its head again!

I reported on a BC anomaly back last October. It intersperses real accesses with one of more UAs Mozilla/4.0 (compatible;) with an HTTP_X_bluecoat_VIA header which, of course, almost no one is set up to recognise so anyone trapping the UA as a hacker (which it often is) then they'd block the access (and probably IP) every time.

Looks like I should add it to my "Don't be so Blue Coat daft!" 403 response message.

Mokita




msg:3860026
 9:56 pm on Feb 28, 2009 (gmt 0)

It is nice to finally know that it is ProxySG causing the 404s, but I won't be doing anything to block or redirect them. I greatly doubt that the person doing the browsing ever sees the 404s, so serving a "Don't be so BlueCoat daft" message is a waste of time.

Just picture this: Everyone in a huge company seeing two 404s before they get to view any web site ... for years on end. They would be beating down the door of their IT department and probably beating the IT staff as well.

[edited by: Mokita at 9:58 pm (utc) on Feb. 28, 2009]

dstiles




msg:3860648
 11:35 pm on Mar 1, 2009 (gmt 0)

I am aware that the general browser won't see the message. Hopefully some IT guy might, at some time - though I doubt it. But the message is already in place as part of my blocking software and I may as well use it. :)

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved