LinkScanner for Dummies
LinkScanner was originally released in 2006, the invention of Roger Thompson of the amusingly named Exploit Prevention Labs. It was easy to exploit then and it is easy to exploit now, but it didn't cause any real problems because almost nobody used it.
Grisoft bought LinkScanner in December 2007, and rebranded as AVG Technologies in advance of the release in April 2008 of version 8.0 of their hitherto popular and successful anti-virus package.
As part of the deal Roger Thompson became AVG's Chief Research Officer, and took all his eighteen staff with him - Greg Mosher became Vice-President of Engineering and Chris Weltzien became Vice-President of Business Development, according to a CNET report.
Problems with AVG LinkScanner first came to light on 31 March when WebmasterWorld contributor Umbra reported that a mystery user-agent (the infamous "1813") was blundering into his security traps. It was a while before DanA linked it to AVG on 28 April - just as users were starting to upgrade in their millions.
As "1813" was clearly a dishonest robotic tool the general response from webmasters was to leave it blocked - for many of us there was no need to do anything, as it already tripped our security traps.
But LinkScanner was so poorly designed that it reacted badly to a straight "403 Forbidden" response, making many repeated requests, and it also came to light that blocking it would mean that AVG users would be discouraged from visiting our sites.
There was a lot of confusion at first, but after a little testing - something AVG Technologies might employ in future - an effective method of dealing with LinkScanner was arrived at on 10 May which would stop it causing problems for webmasters.
Meanwhile the many other deficiencies of LinkScanner were exposed: the colossal waste of bandwidth, the destructive effect on statistical analysis, other poor design features and (most obviously) the fact that as it was so easily fooled it was a security risk for anyone who used it.
WebmasterWorld members contacted AVG and pointed them to our findings.
AVG ignored us.
The Register apparently got involved when alerted by another webmaster whose statistics were going haywire, and had no idea what the full facts were. Journalist Cade Metz was savvy enough to check out WebmasterWorld but understandably (he is not a webmaster) found all the technical debate confusing. He had a worthwhile story, though, and first published on 13 June.
In the article AVG's Roger Thompson gave short shrift to webmasters' concerns and gave The Register his immortal quote: "I don't want to sound flip about this, but if you want to make omelettes, you have to break some eggs."
In the comments to the story AVG asked for webmasters to contact them to help solve the issues. I was probably one of the first to do so and sent a cordial email to Pat Bitton suggesting that WebmasterWorld had all the information AVG needed. The response was arrogant, dismissive, and very close to offensive.
Over the next few weeks AVG happily posted anywhere but WebmasterWorld seeking help. Some websites proudly tell how they were contacted by Roger Thompson himself, and even by AVG's CEO Karel Obluk. The obvious conclusion is that the company had nobody on the payroll who had a clue about the web.
Or about security. The original article in The Register told any malware writer with reading skills exactly how to fool LinkScanner and to safely deliver a drive-by download - if they didn't already know. LinkScanner was so obvious and so easily fooled that anyone could do it.
It seems to have taken quite a while for this to sink in at AVG Technologies, though they were told about it often enough. Eventually they realised that LinkScanner was a security risk and tried to fix it. But they didn't know what they were doing and mistakenly introduced two more obvious fake user-agents before deciding to go back to the original Exploit Prevention Labs method of falsely claiming to be a genuine IE6 user.
Which, of course, was just as easy to fool.
Security companies - naturally - rarely admit to making security gaffes. So AVG's public utterances merely said they wanted to help webmasters with the analytics and bandwidth problems and that "we still enable those webmasters who want to filter our requests out of their results to do so".
But AVG would not tell webmasters how this could be done.
Possibly because they didn't know.
The entire exercise seems to have been a sham. As The Register pointed out this week, if there is any way for LinkScanner to be detected then it remains easy to fool. And The Register proved it by publishing details of one detection method for the whole world to see.
Even the "security experts" at AVG Technologies will understand that method by now, and action to prevent it should be expected in the "Service Pack 1" they have scheduled for release in mid-July.
So where does this leave us?
AVG appear to be saying they allow - by design - methods for concerned webmasters to detect LinkScanner, but will not say what they are and appear to be working hard to make any detection impossible.
Webmasters are busy preparing new methods. They don't like being forced to pay the "LinkScanner Tax" along with their normal bandwidth charges, they don't like having their statistical analysis rendered unusable, they don't like dishonest robots acting like malware to access their sites, and they don't like being insulted by clueless muppets like Roger "The Eggbreaker" Thompson and his friends at AVG.
The Register is also not finished - this sentence in the most recent article caught my eye: