Welcome to WebmasterWorld Guest from 184.108.40.206
But he does tell malware infested drive-by download sites how to fool it.
Imagine the not-so-uncommon scenario where you query Google and some unsavory sites are displayed in the search results, such as adult content, gambling, some of it possibly illegal where you live.
The Link Scanner could quickly access those sites logging your IP as a visitor and create either AUP violations or break laws based on your current location without you even knowing about it.
In this scenario it could easily cost you an ISP, job or worse, put you in trouble with the law. Then trying to explain how you never visited that site when they have proof to the contrary, especially with less than technically savvy types, that should be fun.
This just gets better and better...
For every single one Google search, you additionally get 10 (or 20 /30 /50 /100, depending on your Google SERP settings) unrelated extra visits accounted.
"You are surfing quite a lot over the day, instead of doing your duty work. And why have you been <here> and <there>?"
I've been digesting all the comments about this situation
I just read the LinkScanner comments on The Register and the level of ignorance is astonishing.
I couldn't help noticing, though, that it is only a couple of days since an AVG representative was perfectly happy to post there, and I saw another AVG employee responding about the LinkScanner problem on a technical forum at the weekend.
Both responses were anodyne corporate-speak of the "we're looking into it and will get back to you" variety which (as Jim pointed out) is all that can be expected in the circumstances, but I remain baffled as to why AVG so pointedly snubbed the one site that understood the problem and could actually help them.
Does WebmasterWorld have a reputation for evil that I don't know about?
Does WebmasterWorld have a reputation for evil that I don't know about?
Nah,don't think so, juts the usual - hear no evil, speak no evil, & no personal vendattas
- sure, then there's the I can't dump brain in one thread and expect to be (read as) right// then thered's the ever changing bit where everyne lives and learns - tis the way of the world or this wouldn't happen would it if everyone knew everything?
If my speculation is anywhere near the truth, then LinkScanner 'works', despite significant problems for webmasters, and apparently significant problems for many AVG users. Perhaps they expected fallout, but not quite as much as they received.
And of course, in the absence of any early (or even late!) response to this thread in particular, a large section of participants will appear somewhat hostile. Of course, the way to diffuse this 'hostility' is to address some of people's concerns. This doesn't look like it's going to happen any time soon if the 'breaking eggs' comment is anything to go by ;)
a large section of participants will appear somewhat hostile
What confuses me is you would think they'd feel more at ease to discuss it here as WebmasterWorld has been quite a bit more civilized about this issue than other places that AVG has posted comments.
Perhaps they are misinformed about the number of eyeballs reading WebmasterWorld?
Oh well, with such a reputation management nightmare unfolding, it's just one more snafu to add to the pile!
[edited by: incrediBILL at 8:42 pm (utc) on June 16, 2008]
if the 'breaking eggs' comment is anything to go by
My eggs remain intact - neither AVG LinkScanner nor the Exploit Prevention Labs version even manage to scratch the shells, as both products are so absurdly easy to fool.
Roger Thompson's blog seems quiet though: [blogs.avg.com...]
we were on the forefront of this breaking story with the most accurate information available anywhere
Credit for that is due to all the WebmasterWorld members who contributed in the various (and sometimes confusing) threads about AVG LinkScanner ever since the issue was first raised on 31 March.
Special honours to everyone who tried to contact AVG to help them in those eleven weeks, and to the many who offered suggestions in the forums on what the company needed to do - if AVG had listened they would not be in such a mess now.
As for hostility, AVG should be aware that representatives from Google, Microsoft (arguably the most hated company in the world) and other corporations are happy to post on WebmasterWorld because they know they will get valuable feedback from the people who matter (their customers) and can safely ignore any ranting.
Even if AVG turn up at this late stage I am sure they will be welcomed.
But I doubt if anyone will have a solution for their biggest problem - they have handed a database of their customers' IP addresses to the opposition, and nothing they do will ever get it back.
IncrediBill brings up a VERY VERY valid point, and the more I think about it, the scarier it is.
I've installed AVG Free on probably a hundred machines (for other people), and run paid versions on my stuff.
Bye-bye, AVG...I won't be renewing, and I'll find something safer to install for everyone.
Of LinkScanner designer Roger Thompson the journalist says: "His chief concern is security, and he doesn't want webmasters or malware writers gaming his scanner".
He seems oblivious to the fact that they have been for a month.
Is anyone from El Reg reading this thread?
The journalist in question seems pretty well briefed.
He managed to extract this amusing quote from AVG's Chief Research Officer:
"In order to detect the really tricky - and by association, the most important - malicious content, we need to look just like a browser driven by a human being".
Which, of course is exactly the opposite of what AVG has been doing - LinkScanner identifies itself with a user-agent that no human being ever uses (and no scammer would ever dream of spoofing).
I believe The Register prides itself on its sense of humour.
On the serious side, The Reg has published useful information on the analytics and bandwith issues, warned their readers exactly what to watch out for, and actually managed to get through to three different AVG bigwigs for comment - and they also credited WebmasterWorld and gave us a link.
Six weeks ago I asked "how long should it be before they change the user-agent to something less conspicuous?" and though it must be qualified as speculation at the moment the latest article suggests that "A fix could arrive as early as this week".
This story is not finished yet.
At this point they look like an ignorant bunch of chumps. Eggs no eggs, whatever..
Tha sad part is that scraper scripts are starting to use this UA. I've had to fight off a mini DDOS yesterday originating from OVH and Netdirekt using that User Agent and I know 100% that trafic was not a LinkScanner originated. It is not an issue for this site sinse UA is served nothing but a bunch of random HTML. But then again, I could spot a scraper from a mile away, but it's me... What about the rest of the sites out there?
LinkScanner not only invites them to fool it (which is absurdly easy), but if they redirect it then AVG pays for the bandwidth, and the scammers still get a database of AVG users for future reference.
You couldn't make it up. Which is sad, because the anti-virus product itself is pretty good.
If AVG had any sense - which seems debatable - they would drop LinkScanner completely.
I would stress that the forum is intended for AVG users and not for irate or satirical webmasters. It does, however, contain the official AVG response to the article in The Register (posted yesterday) which includes the following:
"we are issuing previously-planned updates to our free and commercial products this week which will address traffic spike and other issues that have arisen since the first releases of AVG 8.0."
It also contains this (also posted yesterday) from a member of the AVG Team:
"if the LinkScanner is to correctly evaluate the user-threat then it must do it "as" a user, not as an identifiable bot"
Which, as we all know, is the opposite of what it has been doing since it launched.
This week's "pre-planned update" should be interesting.
Testing LinkScanner Accuracy
Out of curiosity I tested LinkScanner against my own homebrew link checker that I use to validate the submitted listings of my directory. My link checker is pretty rudimentary and probably misses some things but it detects enough that I know it's at least protecting visitors from some bad sites, which are then quarantined, that I used for a comparison test.
Sure, the LinkScanner nailed a few of the sites with the invisible IFrame launcher but it failed to flag one of the sites with an IFrame launcher plainly visible in the HTML!
Which my code decoded to:
<iframe src=http://example.com/badthings.html width=1 height=1 style="display:none"></frame>
Sure enough, the code loaded an iframe that loaded a questionable site, that redirected through two sites ending at a site in Russia which then redirected somewhere else, yet it was passed as "safe" because we know the bad sites never cloak.
Anywhere along that redirect chain could be the bad code cloaking good things to the link scanner yet they passed a site that my link checker put in quarantine to protect my visitors.
OK, I'm not a big virus scanning company so what do I know about security?
Here's what I know, I know when I see something bad it should be flagged. I took the source link in that IFrame and did a little research on the web and sure enough, if you send that page with the right parameters it redirected me to a page with a malicious exploit on the page.
I'll give the LinkScanner credit in that it detected a problem when given the right parameters that it too detected the exploit in the destination page but shouldn't the invisible IFrame redirecting through Russia have been enough to flag that site in the first place?
Now to put this thing to the test, I gave the LinkScanner a page from some blog that had spam links to the same malware site and it gave that blog page a clean bill of health.
Would AVG have stopped me from clicking that link or saved me if I did?
I don't know and I'm not going find out, this test is over.
Does LinkScanner Detect Hacked Sites?
Then I ran a few of my sites through the LinkScanner that had been hacked, you know the new SEO hacks where a few hundred spam links are injected into the page?
It said most of those hacked sites, some visibly compromised, were clean except a couple that were flagged as "Link to known exploit site".
Like I said, I'm sure their LinkScanner does a lot more threat detection than my link checker, but it's pretty scary that my little link checking project which has only a few hours of code written dedicated to malware detection was catching and quarantining things their paid product didn't even flag with a warning and gave a green light.
Now I'm wondering what all the fuss is about with the LinkScanner technology as it took less than an hour and I was able to find malicious sites it overlooked and several ways to fake it that have nothing to do with the ";1813" user agent visibility.
[edited by: incrediBILL at 7:12 am (utc) on June 22, 2008]
several ways to fake it that have nothing to do with the ";1813" user agent
You know that. I know that. The "Bad Guys" know that.
As of 20 June (two days ago) Pat Bitton, head of communications at AVG Technologies, is still posting on other sites to say that webmasters can weed out the "1813" user-agent in order to fix their traffic stats.
This was on the same day that the official AVG response to The Register stated that "previously-planned updates" would address the traffic spike issue this week and on the same day that one of Pat Bitton's colleagues stated on AVG's site that LinkScanner must NOT look like "an identifiable bot" (which it currently does and has done ever since it launched).
The phrase "headless chickens" springs to mind, and this whole story would be hilarious if it wasn't for the fact that millions of innocent users are being put at risk by AVG's continuing incompetence - which, given that we have told them all they need to know, also qualifies as negligence.
Here is another quote (posted Friday) from a member of the AVG Team:
My personal take on this issue is that the LinkScanner has introduced a new dimension to the web
That new dimension appears to be corporate suicide.
fact that millions of innocent users are being put at risk by AVG's continuing incompetence
AVG updated yesterday, not sure about the user string. Noticed the link check thing now works in firefox 3 which it did not before.
AVG updated yesterday
The most recent version of AVG Free available for download is dated 5 June.
not sure about the user string
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)
as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk
With respect, you - and AVG - are missing the point entirely.
If you Google any of my sites they will feed LinkScanner a dummy file and get AVG approval.
Many others here take the amusing approach of telling LinkScanner to check AVG's site instead, and they also get the green checkmark and star of approval while AVG gets the bandwidth cost.
In both cases LinkScanner - and YOU - are being comprehensively fooled by the good guys.
The "Bad Guys" also know how to perform this simple task and the exploit filled websites you mention are under THEIR control, so LinkScanner - and YOU - will be fooled by them also.
Enjoy your drive-by, but don't say you were not warned.
as opposed to being put at risk of incompetent webmasters and exploit filled websites, I'll take the AVG risk.
I think you missed my post 4 above this, I found multiple instances where it green-lit some of the nastiest sites I've ever seen. When presented with a live invisible IFrame script it followed all the redirects and claimed it was GREEN yet give it a slightly different parameter so it could follow redirects to an explicit malware path, only then claimed it was RED.
It's definitely a risk when it can see the fingerprint of malware yet errs on the side of caution and says the site is OK when there is no valid purpose for that invisible IFrame and that type of code doesn't exist in the tens of thousands of sites I've scanned other than when those sites are compromised.
Yup, enjoy the risk because it's one I wouldn't want to take.