homepage Welcome to WebmasterWorld Guest from 23.20.77.156
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 219 message thread spans 8 pages: 219 ( [1] 2 3 4 5 6 7 8 > >     
Register Scolds AVG For Generating Fake Traffic As Link Malware
Webmasters Complain AVG Debilitating Traffic Analytics
Samizdata




msg:3674412
 8:52 pm on Jun 13, 2008 (gmt 0)

In an otherwise interesting article about AVG LinkScanner the author spectacularly misses the point that because it can easily be identified it is worse than useless as a security tool.

But he does tell malware infested drive-by download sites how to fool it.

[theregister.co.uk...]

...

 

incrediBILL




msg:3674532
 12:37 am on Jun 14, 2008 (gmt 0)

This is a new chapter in an ongoing thread about AVG started on 3/31/08 right here on WebmasterWorld
[webmasterworld.com...]

Love this quote from Roger Thompson, CTO of Exploit Prevention Labs:
"There's so much hacking activity going on the web. The only way to really tell what's there is to go and have a look," he told us. "I don't want to sound flip about this, but if you want to make omelettes, you have to break some eggs."

Which eggs are they trying to break?

IMO, when your anti-hacking tool activity is completely visible to the hackers it's all the AVG customers eggs that are in jeopardy.

However, we've been round and round with this and now it's in the press yet AVG seems not only to not care but is giving flippant quotes.

There are better methods that don't result in any damage being inflicted on bookmarked websites. For instance, one of their competitors has a better solution which involves a transparent proxy and real-time streaming data analysis. It doesn't expose itself to the world, websites don't know it's there, and it still catches the bad sites without spiking traffic needlessly on unsuspecting innocent websites.

[edited by: incrediBILL at 1:24 am (utc) on June 14, 2008]

g1smd




msg:3674539
 12:43 am on Jun 14, 2008 (gmt 0)

AVG are interested in security. They want to scan your site before they rate it as "visitable" for their users.

Webmasters see malicious bots attempting to access their sites on a regular basis. Anything out of the ordinary, or not directly from Google, Yahoo, Live, Ask, is likely to be blocked.

So, where the middle-ground?

incrediBILL




msg:3674546
 12:51 am on Jun 14, 2008 (gmt 0)

AVG are interested in security. They want to scan your site before they rate it as "visitable" for their users

Scan it once a day, 1-Hit from any AVG installation, just once. Store the results on AVG's servers and let the rest of the AVG users get the results from AVG's cache.

What they've done instead is generate a virtual DDoS on some servers that have sites with many bookmarks, 10's of thousands of people bookmark some sites, maybe millions.

So what do you think happens when all those thousands of AVG 8 customers link scan?

That's right, you get a DDoS when all those prior visitors that have your site bookmarked install AVG 8.0!

The only upside is that AVG isn't installed on as many machines as the top AV software or we'd all be in trouble already.

[edited by: incrediBILL at 12:54 am (utc) on June 14, 2008]

Samizdata




msg:3674547
 12:52 am on Jun 14, 2008 (gmt 0)

AVG are interested in security

Then they should read WebmasterWorld and not The Register.

...

The article says:

1. AVG LinkScanner can be detected by user-agent (details supplied).

2. It can be fooled by feeding it a dummy file (example site given).

3. Further details are available on WebmasterWorld (for those who need them).

I suspect that most malware distributors already know this.

...

The article does not say:

1. Those who rely on AVG LinkScanner for security are being misled.

2. Any malicious site can fool it and deliver them to a drive-by download page.

3. LinkScanner is actually a security risk and makes AVG look incompetent.

I suspect that Grisoft don't understand this yet.

...

[edited by: Samizdata at 12:56 am (utc) on June 14, 2008]

g1smd




msg:3674565
 1:15 am on Jun 14, 2008 (gmt 0)

*** 1-Hit from any AVG installation, just once. Store the results on AVG's servers and let the rest of the AVG users get the results from AVG's cache. ***

I think that would bring up some privacy concerns. By using log files etc, AVG could, in theory, find out which sites a particular user had visited.

incrediBILL




msg:3674575
 1:38 am on Jun 14, 2008 (gmt 0)

You don't need to store the source of the scan, just the results of the link scan, privacy can be maintained.

However, this has already created some privacy concerns IMO as many of us now have very large lists of AVG customers by logging all of the bogus user agents. Use your imagination in what could be done if AVG has any gaps in their protection.

If/when AVG switches their software to use the actual user agent, to avoid being detected, we still have that big list of users coming from all those locations so we'll still know where a large majority of AVG users exist on the internet. So simply fixing the user agent won't help anyone who's AV software preference has already been exposed.

At a minimum, I'm sure some other AV software makers or their affiliates would love this targeted list and have probably already started the conversion process.

[edited by: incrediBILL at 1:39 am (utc) on June 14, 2008]

Brett_Tabke




msg:3674612
 3:33 am on Jun 14, 2008 (gmt 0)

> lists

That was going to be my suggestion - a public list of all AVG customer IP's.

Samizdata




msg:3674616
 4:11 am on Jun 14, 2008 (gmt 0)

a public list of all AVG customer IP's

Google beat you to it Brett - try a search for:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

...

Samizdata




msg:3674647
 7:36 am on Jun 14, 2008 (gmt 0)

Somebody named Pat Bitton from AVG responded to the article in the comments:

Over the next few days, we will be exploring ways in which we can continue to deliver informed protection as unobtrusively as possible without adversely impacting site analytics. Any webmaster reading this post who is interested in working with us constructively to reach this goal is welcome to contact me

My offer of assistance was rebuffed.

...

Staffa




msg:3674664
 8:05 am on Jun 14, 2008 (gmt 0)

Roger Thompson earlier this week, he was unaware of these issues. But he defended the role of Linkscanner, which he designed...
... he said. "Given that we've only just been alerted to this situation...

Mr Thompson, when you design something your first requirement is to test it thoroughly before releasing it to the world.

Staffa




msg:3674667
 8:12 am on Jun 14, 2008 (gmt 0)

Samizdata, don't feel the worse for it.
If he means what he says he would be posting here but I guess that's too much of a challenge for him.

gaouzief




msg:3674673
 8:38 am on Jun 14, 2008 (gmt 0)

*** 1-Hit from any AVG installation, just once. Store the results on AVG's servers and let the rest of the AVG users get the results from AVG's cache. ***

That would be too exensive, you need to maintain a bot, storage, bandwith etc..., right now they are using their client's ressources for this "feature"

Samizdata




msg:3674678
 8:51 am on Jun 14, 2008 (gmt 0)

If he means what he says he would be posting here

I advised him/her to do so and he/she has certainly read this thread.

...

incrediBILL




msg:3674682
 9:07 am on Jun 14, 2008 (gmt 0)

The comments on The Register are mostly negative about AVG as this Link Scanner appears to be having some serious impact on the performance of people's machines using it as well as the web sites that are being abused.

It's a nice 2-for-1 special, AVG users and webmasters all irritated for one low price.

incrediBILL




msg:3674684
 9:14 am on Jun 14, 2008 (gmt 0)

That would be too exensive, you need to maintain a bot, storage, bandwith etc..., right now they are using their client's ressources for this "feature"

No, a bot can be easily spotted, won't work, trust a bot spotter about this. Using the client machine for a distributed crawl of these sites is much harder to detect and the best way to compile an accurate list of malware sites that isn't easily spoofed.

However, if AVG 8 can actually defend against the malware that the Link Scanner detects then the Link Scanner itself just wasting everyone's time and this feature is just marketing hype with no intrinsic value.

jdMorgan




msg:3674764
 2:46 pm on Jun 14, 2008 (gmt 0)

It would be very helpful if the client-side LinkScanner simply cached the scanned page contents for any given Search Engine Results Page scan. In this way, our sites are not hit repeatedly each time the user returns to the results page.

As it is now, a user does a search and gets (typically) ten results. LinkScanner loads those ten results URLs and analyzes them. Now if the user clicks on one of the links in the results, and then later comes back to the same results page, those ten results are scanned again. If yours is the last result on the page and the user goes through and clicks each result in turn, returning to the results page each time, then your page gets loaded and scanned ten times. And this can be even worse if your site has multiple listings in that one results page.

Some have said that they have to do it this way so that the scan results are fresh and up to date. I disagree and would like to point out that the scan is done just after the search results page loads. There is no guarantee that the user will immediately click on any of the link(s) on that results page that he/she is interested in, so some degree of latency is already present in the system.

The main problem they appear to face is the LinkScanner is a "side-car" to the browsing process, rather than a in-line operation. That is, it appears to be part of the Anti-Virus program rather than a browser plug-in. A plug-in approach might make it easier to take advantage of client-side caching, and to use the browser's user-agent instead of the easily-identifiable ";1813" with an IE User-agent.

As stated in one of the other threads here, I'm grateful to AVG for providing a good and free AV solution to those who can't afford to spend $25-$50 per year on AV subscriptions. So let's not beat them up too hard on this until we find out if they will take our interests into consideration now that the problem has been made known. The simple fact is that the additional load on servers, pollution of site statistics, and the security implications of the easily-detected and easily-spoofed user-agent just were not thoroughly considered.

As for LinkScanner building/maintaining a database of bad sites to be kept on a centralized server, that can also be done on a voluntary basis to alleviate privacy concerns -- Let the user opt-in to the reporting and checking feature just like many other such services do...

Jim

incrediBILL




msg:3674790
 3:22 pm on Jun 14, 2008 (gmt 0)

It would be very helpful if the client-side LinkScanner simply cached the scanned page contents for any given Search Engine Results Page scan. In this way, our sites are not hit repeatedly each time the user returns to the results page.

That won't solve the sheer volume of hits I'm seeing from thousands of different IPs on a daily basis. It's early in the morning on a slow traffic weekend and AVG's code has already generated over 1K unique IPs hitting the site.

The main problem they appear to face is the LinkScanner is a "side-car" to the browsing process, rather than a in-line operation.

Jim, I respectfully disagree as the main problem is it doesn't need to exist in the first place!

This is an instance where the technology implemented was brute force approach and it's causing major issues for both their customers (see comments on The Register) and webmasters.

My AV program does the same exact page scanning (not link scanning) except it does it in real time using a transparent proxy between my browser and the internet. If something is detected in the data stream it's blocked in real-time, no need to link scan whatsoever, completely transparent.

Why doesn't AVG simply do the same thing and stop annoying everyone?

[edited by: incrediBILL at 3:23 pm (utc) on June 14, 2008]

Peter




msg:3674798
 3:35 pm on Jun 14, 2008 (gmt 0)

I see three points in my logs which may not have been made clear.

First, AVG is too dumb to use compression, which means it's taking even more bandwidth.

Secondly, AVG doesn't only precharge the page that Google proposes (example.com/interesting.html), but ALSO the site index page (example.com/) - even though Google does not propose / in the results in question.

Thirdly, AVG often (but not always) doesn't understand example.com/interesting/ and leaves off the final /, and so gets a 301 that it follows.

These log entries show the first two points, AVG downloads the page that Google proposes and the site index page without compression, then the user dowloads the page that Google proposed with compression:

90.16.11c.dd - - [14/Jun/2008:08:22:29 +0200] "GET /aaa/bbb/interesting.html HTTP/1.1" 200 46506 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
90.16.11c.dd - - [14/Jun/2008:08:22:29 +0200] "GET / HTTP/1.1" 200 8365 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
90.16.11c.dd - - [14/Jun/2008:08:22:31 +0200] "GET /aaa/bbb/interesting.html HTTP/1.1" 200 15655 "http://www.google.fr/search?hl=fr&q=%22blabla%22&btnG=Recherche+Google&meta=" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)"

Here are the logs for the third point (the 301), which may only happen with Yahoo:

79.88.25c.dd - - [13/Jun/2008:10:36:08 +0200] "GET /aaa/interesting HTTP/1.1" 301 259 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
79.88.25c.dd - - [13/Jun/2008:10:36:08 +0200] "GET /aaa/bbb/other.html HTTP/1.1" 200 84606 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"
79.88.25c.dd - - [13/Jun/2008:10:36:08 +0200] "GET /aaa/interesting/ HTTP/1.1" 200 25936 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)"

Peter

PS. [Edit] This suggests that a 301 to AVG's own site would solve the ;1813 problem nicely, but I haven't tried it!

[edited by: Peter at 3:41 pm (utc) on June 14, 2008]

wilderness




msg:3674836
 5:03 pm on Jun 14, 2008 (gmt 0)

This suggests that a 301 to AVG's own site would solve the ;1813 problem nicely, but I haven't tried it!

I've had one in place since this tongue-in-cheek mention:
[webmasterworld.com...]

It hasn't slowed the requests down a bit, nor has the 301 eliminated successive requests for the same page.

Samizdata




msg:3674930
 9:24 pm on Jun 14, 2008 (gmt 0)

let's not beat them up too hard on this until we find out if they will take our interests into consideration

I have no interest in beating up AVG - but I can't help laughing at them.

WebmasterWorld members have been highlighting the LinkScanner problems for more than a month, and some have tried to help the company by contacting them directly - only to be met with complacency (at best) or something very close to offensiveness.

I sincerely hope that others who respond to Pat Bitton's cry for help can get through to them.

Even if the company ignores our interests, they surely need to consider their own.

...

keyplyr




msg:3674994
 11:56 pm on Jun 14, 2008 (gmt 0)

As stated in one of the other threads here, I'm grateful to AVG for providing a good and free AV solution to those who can't afford to spend $25-$50 per year on AV subscriptions - jdMorgan

Just a FYI - AVG is no longer free, that's over with. All the free AVG accounts are expiring and being prompted to purchase various levels of software protection.

jdMorgan




msg:3674995
 12:04 am on Jun 15, 2008 (gmt 0)

Um, no. Try free dot avg dot com. It's true they promote the paid "upgrade" heavily, make the link to update the free version a bit hard to find, and restrict the free version to private, non-commercial use, but it's still there. In fact, I installed the free version on a machine to test the problems described in this thread.

Jim

keyplyr




msg:3675102
 5:32 am on Jun 15, 2008 (gmt 0)

OK, but to elaborate...

My own free copy (single machine, non-commercial) of AVG would no longer download virus defs, announcing that the free version had been discontinued and that to continue to protect my blah blah I would need to upgrade.

I wasn't aware I could have downloaded/installed another copy. Betcha it turns into a bait'n switch.

Samizdata




msg:3675284
 2:56 pm on Jun 15, 2008 (gmt 0)

Betcha it turns into a bait'n switch

What is has turned into is a farce.

Like many other packages AVG offers a free version and uses it to encourage people to buy the paid version - nothing wrong with that, and AVG anti-virus itself works very well and has deservedly become very popular.

Whether the popularity will survive the LinkScanner fiasco remains to be seen.

If AVG's "head of egg-breaking" keeps his job it won't deserve to.

Our primary responsibility is to provide the best possible protection for our users

Mr Thompson, identifying the IP addresses of AVG users in every site log on the web - some of which are publicly available - is a gross dereliction of your primary responsibility and you should publicly apologise for it.

In addition, LinkScanner is so easy to fool that it is the very opposite of a security tool, and in promoting it as "protection for our users" your company is not only selling snake-oil but is also putting its customers at risk.

I don't know what Grisoft paid for LinkScanner or what they are paying you, but they would have been much better off putting all the cash into a big pile and setting fire to it.

...

Seb7




msg:3675388
 6:29 pm on Jun 15, 2008 (gmt 0)

Redirect the toolbar agent to the AVG home page might get things moving.

Samizdata




msg:3675656
 4:04 am on Jun 16, 2008 (gmt 0)

According to Thompson, nearly all web exploit toolkits track IP addresses

According to Samizdata, scammers will now have the IP addresses of many AVG users.

Perhaps someone who still has a job at AVG can confirm this?

Even a "no comment" would be civil.

...

[edited by: Samizdata at 4:11 am (utc) on June 16, 2008]

jdMorgan




msg:3675683
 5:12 am on Jun 16, 2008 (gmt 0)

keyplayer,

> My own free copy (single machine, non-commercial) of AVG would no longer download virus defs, announcing that the free version had been discontinued and that to continue to protect my blah blah I would need to upgrade.

The most likely cause for this is that you had AVG 7.x. They announced (last year?) that 7.x would no longer be supported, and that users should "update" to 8.x (free version also available, if somewhat obscure).

Now, whenever you go to "update," the first thing you see on the page is their offer to "upgrade" to the full-featured paid version. You always have to scroll to the bottom, or even scroll to the right to find the "update" link for the free version. Just an example of marketing at work here, but once you know how it works it's no longer any bother. IIRC, ZoneAlarm does the exact same thing whenever there's an update to their free firewall.


Samizdata,

You are unlikely to get Mr. Thompson, AVG, or any other major Web company to publicly acknowledge any serious problems such as the ones we've identified here. For legal reasons, it's simply not done -- at least not until the corporate legal and marketing departments have approved a statement. Having been paid by AVG for the LinkScanner technology, it's fairly easy to understand Mr. Thompson's position here.

We've raised the flag, we've rung the bell, we've shouted the alarum. We'll just have to wait and see what they do now that the problems have been identified and various solutions/improvements proposed. It will undoubtedly take quite some time...

Jim

Samizdata




msg:3675691
 5:41 am on Jun 16, 2008 (gmt 0)

Wise words as usual Jim, and I am aware of the corporate legalities.

But 48 hours have passed since AVG told me there was nothing in this thread that interested them and I was wondering whether they might have changed their opinion at all in that time.

I will, however, take your advice and wait for the official statement.

...

rise2it




msg:3675731
 7:41 am on Jun 16, 2008 (gmt 0)

"told me there was nothing in this thread that interested them"

Maybe they'd be interested to know I'm getting ready to drop about two dozen paid subscriptions over this crap.

Yes, you can turn it off, but then you get to look at an ugly logo in your try because you turned this off and aren't "fully" protected.

This 219 message thread spans 8 pages: 219 ( [1] 2 3 4 5 6 7 8 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved