homepage Welcome to WebmasterWorld Guest from 54.211.181.45
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 219 message thread spans 8 pages: < < 219 ( 1 2 3 4 5 6 [7] 8 > >     
Register Scolds AVG For Generating Fake Traffic As Link Malware
Webmasters Complain AVG Debilitating Traffic Analytics
Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 8:52 pm on Jun 13, 2008 (gmt 0)

In an otherwise interesting article about AVG LinkScanner the author spectacularly misses the point that because it can easily be identified it is worse than useless as a security tool.

But he does tell malware infested drive-by download sites how to fool it.

[theregister.co.uk...]

...

 

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3674410 posted 1:05 am on Jul 5, 2008 (gmt 0)

I log submitted form content plus environment variables for anti-form-spam traces if the browser parameters look dodgy (missing referers etc). Looking through one such log I find a variety of actual dummy characters: typically underscores, hyphens, tildas and X's.

The format of the dozen or so I've just examined suggests the blanks to be HTTP_ACCEPT_ENCODING and HTTP_REFERER - the character counts fit, the latter's value is a variable length and both variables always seem to be otherwise missing.

I've always put it down to some half-baked anti-virus / firewall app that's trying to hide stuff that it deems unimportant just so it can claim to be protecting the ignorant victim - sorry, user! None of the important stuff ever seems to be missing apart from those two - which, of course, turns the user into a rejected robot by incredibill's logic.

I do know that the dummy/missing params sets are often in valid browsers: I can determine this from the contents of the trapped forms, which are obviously different from auto-submitted spam.

mlduclos

5+ Year Member



 
Msg#: 3674410 posted 1:44 am on Jul 5, 2008 (gmt 0)

Hello, here is my answer from AVG, probably an email template:

<snip>No emails please, provide a synopsis</snip>

[edited by: incrediBILL at 9:47 am (utc) on July 5, 2008]
[edit reason] TOS #9 - No Email Excerpts [/edit]

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 1:59 am on Jul 5, 2008 (gmt 0)

AVG's primary responsibility is to protect our end-users

AVG are clearly failing in their primary responsibility.

Instructions on fooling LinkScanner are all over the web (including The Register and Slashdot).

To protect their end-users AVG should dump LinkScanner in the trash where it belongs.

Anything less would be irresponsible.

...

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3674410 posted 3:14 am on Jul 5, 2008 (gmt 0)

Excuse me if this has been mentioned previously in either of the various marathon threads concerning AVG.

Up until now I have blocked the UA "User-Agent:" since 99% of the time it is used with SPAM attempts on my input forms. But in the last couple days I've noticed a majority of the HEAD requests coming from AVG now use "User-Agent:"


HEAD / HTTP/1.1" 403 0 "-" "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Needles to say, I made haste to remove this block across all the sites I manage. So are we now victim to the whim of AVG every time they decide to alter their UA?

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3674410 posted 3:26 am on Jul 5, 2008 (gmt 0)

keyplr,
I've left the previous denial for "begins with User-Agent" in place, however placed it behind other rules.
It's not flawless, nor without error, however it works.

Don

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3674410 posted 4:07 am on Jul 5, 2008 (gmt 0)

keyplyr,

If it starts with "User-Agent:" and ends with "SV1)" and has the other details IncrediBill posted above, it is likely to be LinkScanner.

If it starts with "User-Agent:" without ending with "SV1)", it is likely your old form spammers.

Jim

Mokita

5+ Year Member



 
Msg#: 3674410 posted 5:00 am on Jul 5, 2008 (gmt 0)

AVG have responded twice today to concerns being aired on an Australian forum (the italics for emphasis is my addition):

Firstly:
[forums.whirlpool.net.au...]

I can categorically assure every member here that AVG has heard you, and that we fully understand the issues that have been raised here and that this message has been loudly communicated to the AVG Technologies development teams in both the USA and Europe. I assure you that you currently have the attention of the Chief Technology Officer who is the person responsible for the design and development of the AVG product range.

As a matter of urgency, AVG is evaluating the LinkScanner technology so that we can ensure we maintain the level of protection for AVG users, but to minimise the impact on web sites and minimise any bandwidth overheads to both AVG users and web hosts.

And secondly:

[forums.whirlpool.net.au...]

In working with the web master community, AVG has responded immediately and on Tuesday, July 9th, AVG will issue a product modification to address the spikes that a few individuals have seen with their web traffic.

We have modified the Search-Shield component of the product to only notify users of malicious sites. Search-Shield no longer scans each search result online for new exploits, which was causing the spikes that web masters addressed with us. However, it is important to note that AVG still offers full protection against potential exploits through the Active Surf-Shield component of our product, which checks every page for malicious content as it is visited, but before it is opened.

[edited by: Mokita at 5:06 am (utc) on July 5, 2008]

[edited by: incrediBILL at 8:12 pm (utc) on July 5, 2008]
[edit reason] trimmed quotes and clean up formatting [/edit]

Scarecrow

10+ Year Member



 
Msg#: 3674410 posted 5:59 am on Jul 5, 2008 (gmt 0)

Frankly, Borrett's message sounds like more spin to me. The test will be whether the ratio of LinkSpanner to eyeball hits goes down after July 9. While they probably have some auto-update feature that will help propagate the revision, I doubt that it will be very effective. Maybe we'll see improvement after several weeks.

What they'll have to do is offer a public apology and a public recall of the LinkScanner component in their product, so that the story is picked up by the major media. That will get the word out, and then they can work on a new approach with less hype, and no mass prefetching, once the dust settles.

By then everyone will be using some other type of software.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3674410 posted 8:07 am on Jul 5, 2008 (gmt 0)

I've left the previous denial for "begins with User-Agent" in place, however placed it behind other rules. It's not flawless, nor without error, however it works. -Don

If it starts with "User-Agent:" and ends with "SV1)" and has the other details IncrediBill posted above, it is likely to be LinkScanner. If it starts with "User-Agent:" without ending with "SV1)", it is likely your old form spammers. - Jim

Thanks guys :)

Yeah, even though I have CAPTCHAs I still need those "User-Agent:" rules or I get hammered so I came up with a rewrite that lets the LinkScanner through. But for the record, I have done so under duress.

It will interesting to see how this AVG problem plays out.

Mokita

5+ Year Member



 
Msg#: 3674410 posted 9:20 am on Jul 5, 2008 (gmt 0)

Scarecrow wrote:
Frankly, Borrett's message sounds like more spin to me.

Both messages are 99% spin to me.

While they probably have some auto-update feature that will help propagate the revision, I doubt that it will be very effective. Maybe we'll see improvement after several weeks.

They do have an auto-update for both the free and paid versions - why do you doubt it will be effective? If it stops AVG scanning all unclicked links on pages of SERPS as they are promising, that is a good result for webmasters.

What they'll have to do is offer a public apology and a public recall of the LinkScanner component in their product, so that the story is picked up by the major media. That will get the word out, and then they can work on a new approach with less hype, and no mass prefetching, once the dust settles.

Huh? There is no way they will voluntarily compromise their already severely, battered reputation by offering a public apology or recall in the major media. They will try to avoid that at all costs.

By then everyone will be using some other type of software.

True for those who are net savvy, or who have advisors (family, friends, computer repair shops etc) that are aware. But the greatest number of the computer-illiterate users of AVG will continue on, completely and blissfully none-the-wiser.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3674410 posted 10:06 am on Jul 5, 2008 (gmt 0)

Search-Shield no longer scans each search result online for new exploits, which was causing the spikes that web masters addressed with us.

This is the key statement which sounds to me like LinkScanner as we know it, is one dead puppy.

What I'm reading into this which makes sense is that the Search-Shield will use stale results just like the Site Advisor does, and the Active Surf-Shield will probably share the results of new infected sites it encounters with AVG to update Search-Shield.

This is what it should've been in the first place and now, thanks to a large dose of reputation management problems, what ends up probably being a really good solution will have already been so massively tarnished that it won't get the recognition it deserves now that it's going to behave reasonably.

Guess we'll have to wait and see what happens on the 9th.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3674410 posted 10:11 am on Jul 5, 2008 (gmt 0)

sorry, user! None of the important stuff ever seems to be missing apart from those two - which, of course, turns the user into a rejected robot by incredibill's logic

Sorry to disappoint you but I'm not the only person that blocks based on bad headers and up until this AVG thing hit the web, I blocked maybe a handful of IPs daily (10-20) with this problem, which is statistically insignificant out of 20K+ visitors a day.

Maybe you found a rogue case of something that appeared to be human but I have never seen a real human on my site browsing without an Accept header and if they are, that's a real shame because there's a limit to how much malfunctioning garbage I'll allow to abuse my site.

FWIW, It was that logic that snared this AVG LinkScanner so it obviously was a good thing I had it hardwired to deal with the situation in the first place.

[edited by: incrediBILL at 10:11 am (utc) on July 5, 2008]

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 10:57 am on Jul 5, 2008 (gmt 0)

this message has been loudly communicated to the AVG Technologies development teams

We've received the following communication from AVG Technologies HQ

While waiting for the 9 July changes I would like to express some respect for Peter Cameron and Lloyd Borrett of AVG Australia/New Zealand - while they are (and must be) constrained in what they say they are clearly not afraid to communicate with the antipodean webmaster community.

Frankly, Borrett's message sounds like more spin to me

The spin was in the message from AVG HQ that Mr Borrett kindly passed on.

Neither of these gentlemen deserves criticism in my opinion.

...

Receptional Andy



 
Msg#: 3674410 posted 11:12 am on Jul 5, 2008 (gmt 0)

>> only notify users of malicious sites

Presumably this means there will only be red flags now, so blocking linkscanner entirely won't cause any issues with SERPs?

I also note that on the download page for AVG there is now a 'linkscanner database', which hints at some kind of centralisation.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 11:46 am on Jul 5, 2008 (gmt 0)

The LinkScanner database has always been part of the program - I assume it contains a list of blacklisted IP addresses, and that is apparently why one WebmasterWorld member on a shared IP found his site wrongly banned.

Luckily he found out and managed to get delisted, but it wasn't AVG that told him.

[webmasterworld.com...]

...

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 11:57 am on Jul 5, 2008 (gmt 0)

In working with the web master community, AVG has responded immediately and on Tuesday, July 9th, AVG will issue a product modification to address the spikes that a few individuals have seen with their web traffic.

I will not bother to point out that AVG has been conspicuously avoiding working with this particular webmaster community and that their response to the problem was the very opposite of immediate.

But according to my calendar 9 July is a Wednesday.

...

Mokita

5+ Year Member



 
Msg#: 3674410 posted 12:26 pm on Jul 5, 2008 (gmt 0)

AVG have apparently already rolled out the update in their download for new installations of the free version. Those of you who enjoy testing can get a look at the changes:

[forums.whirlpool.net.au...]

G'day,

Last night AVG Technologies released an update to AVG Anti-Virus Free Edition 8.0 to address the LinkScanner issue amongst other things.

The file name you want to use is:
avg_free_stf_en_8_138a1332.exe (or later)

If you don't want to wait for the update process to roll-out to your PC, or you're doing a new installation, then the up-to-date version is now available for you to download at [avgfree.com.au...]

Best Regards, Lloyd Borrett
Marketing Manager, AVG (AU/NZ)


Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 1:06 pm on Jul 5, 2008 (gmt 0)

I just tested the 4th July release against two of my sites.

To the user, everything looks the same - green checkmark of approval.

To the webmaster, everything looks different - no entry in the logs whatsoever.

The roll-out will take a while so expect to see the AVG user-agents for a few days, but first impressions suggest that the much-vaunted "LinkScanner technology" just bit the dust.

This should be a relief for AVG users who will no longer be endangered by it.

...

Scarecrow

10+ Year Member



 
Msg#: 3674410 posted 1:41 pm on Jul 5, 2008 (gmt 0)

Question: How can this new version look the same to the user? You can't show a green checkmark unless you check the site. Are they checking it against their own database before they show a green check?

At a minimum, they'll have to go through their massive pile of old press releases where they claim that LinkScanner is the only "real-time" checker of all the links from a search engine. It's not "real-time" if they're checking a database.

And you can't show a green mark until you check something. That would be deceptive.

This centralized database thing is risky too. What happens when it is difficult to access? Does everything stay gray? That is something the user will be sure to notice.

I guess I'm too lazy to download the new revision, but I'm not clear on what the green check shown to the user really means in this new version.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 2:08 pm on Jul 5, 2008 (gmt 0)

I'm not clear on what the green check shown to the user really means in this new version

It means that AVG do not know if the link leads to a drive-by download. So no change there.

But if, as seems likely, AVG have finally seen the error of their ways and done something about it then they should not be criticised for it, despite all the corporate gloss they put on the events.

I would still advise any AVG user with a static or infrequently-changing IP address to find
another anti-virus package, but for obvious reasons it is unlikely that AVG will do the same.

...

Scarecrow

10+ Year Member



 
Msg#: 3674410 posted 2:16 pm on Jul 5, 2008 (gmt 0)

If the green check means that AVG doesn't know anything, that is a huge change. In the version of the free LinkScanner I tested 10 days ago, the links from a Google search came up gray. Then LinkScanner fetched each link using multi-threading, and they turned green. If you had 100 links per Google results page, you could see them turning green because it took about 15 seconds or more to go through them.

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 2:37 pm on Jul 5, 2008 (gmt 0)

If the green check means that AVG doesn't know anything, that is a huge change.

Not on my sites it isn't. Same for most of us here. And for malware writers.

That is why AVG had to dump LinkScanner.

...

blend27

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 3674410 posted 6:26 pm on Jul 5, 2008 (gmt 0)

-- This centralized database thing is risky too. What happens when it is difficult to access? --

I hope the woun't use the DB that was populated by LinkScanner, cause it is pretty useless at this time, e.g. it was very easy to game...

Scarecrow

10+ Year Member



 
Msg#: 3674410 posted 7:52 pm on Jul 5, 2008 (gmt 0)

I installed the latest free AVG 8, the one with the filename that ends in 1332.exe. The install screen says it is Build 138, 7/4/2008.

I asked for a custom install, and requested LinkScanner. After installation, the main program screen said that everything, including LinkScanner, was active.

The upshot is that the page scanning is disabled but it still goes through the motions. For the user, everything looks like it did before. All they're doing now is the DNS lookup. If you mouseover the geen check mark, it says "Safe: This page contains no active threats," and gives the IP address and the date and time, and the number of seconds it took to scan this page. However, no scan was done at all on my sites that I was monitoring simultaneously with a tail -f on my access_log. Only the DNS was done, judging from the fact that all green checks had the IP address behind the mouseover.

I guess this means that everyone gets all-green in this version! Maybe there's some database check now, but I rather doubt it. It looks like they commented out the calls to the page scan and analysis and recompiled the package, and that's about all.

This is an emergency "fix" that merely skips the page scans. They will have to redesign their interface and purge a lot of LinkScanner stuff from their publicity as the next step. You cannot claim that you are scanning a page when you merely do a DNS lookup from the user's machine. There are laws against deceptive advertising.

Look for an official statement from AVG about their new "approach" to LinkScanner. If they drag it out in hopes of collecting more cash in the meantime, I'm sure their lawyers will remind them that this is naughty.

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 3674410 posted 8:09 pm on Jul 5, 2008 (gmt 0)

Incredibill:
"...browsing without an Accept header..."

It's the HTTP_ACCEPT_ENCODING one that's missing (or did you mean that one?), along with the REFERER. Definitely valid visitors and not only in forms: they wander through a site the same as humans and look entirely human. We get several on some types of site - oddly, a lot from legal people booking training courses.

The Accept, Language and Charset ones are usually present.

In any case your combination would not kill the injection spammers. I'm currently killing hundreds a day on our small-ish server that are coming in with bare-bones 50727 UAs with ACCEPT, ENCODING and LANGUAGE set and CHARSET and REFERER missing (CHARSET is often missing).

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3674410 posted 8:22 pm on Jul 5, 2008 (gmt 0)

OK, typo on my part.

Actually after allowing things I whitelist, I check HTTP_ACCEPT to see if it's BLANK, "text/html, text/plain" or "text/html" and any of those gets the boot which was sufficient to nail LinkScanner.

incrediBILL

WebmasterWorld Administrator incredibill us a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month



 
Msg#: 3674410 posted 9:58 pm on Jul 5, 2008 (gmt 0)

As have mentions it looks like AVG has a fix from July 4th that no longer assaults sites whatsoever:
avg_free_stf_en_8_138a1332.exe

You must download the file directly from their site, not Download.com or anywhere else:
[free.avg.com...]

The best part is their FAQ page about this fix is a complete fiction.

[free.avg.com...]
Program update AVG 8.0.138
Fixed Bugs

* Fixed problem with link scanning if <base> tag is presented on a web page.

Now that FIX description alone is enough to kill what little remained of their credibility.

Would you use an AV product that blatantly lied?

What else are they hidding?

[edited by: incrediBILL at 10:04 pm (utc) on July 5, 2008]

Samizdata

WebmasterWorld Senior Member 5+ Year Member



 
Msg#: 3674410 posted 10:05 pm on Jul 5, 2008 (gmt 0)

While we are asking questions:

What do you call a LinkScanner that doesn't scan links?

Anyone fancy an omelette?

...

jdMorgan

WebmasterWorld Senior Member jdmorgan us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 3674410 posted 10:41 pm on Jul 5, 2008 (gmt 0)

It is quite possible that the purported "DNS lookup" mentioned previously is done using servers at AVG, which then either reference previously-collected data, or issue (or queue) a new request with a perfect browser spoof if that data is stale -- however they may now define "stale."

If their legal and PR departments are any good at all, you'll never see any public admission that anything was really wrong. That is a fact of corporate survival instinct, and is to be expected.

I think our collective energy would best be spent analyzing they're doing now, rather than casting aspersions. I submit that what we (as Webmasters) need concern ourselves with in this thread is whether this quasi-DOS tapers off as users update to the latest version, and whether their "new method" has any further negative effects on our servers.

Jim

Scarecrow

10+ Year Member



 
Msg#: 3674410 posted 10:52 pm on Jul 5, 2008 (gmt 0)

If they don't come clean and tell the truth to the press, heads will eventually roll at AVG. That's one way to solve the "egg on your face" problem.

However, they could get away with it. It's always been my impression that the anti-virus market is based about 90 percent on lies. I blame the media mostly.

"Your horse ran away and you haven't seen it since? Buy our latch for your barn door and that same horse won't run away again!"

Scarecrow

10+ Year Member



 
Msg#: 3674410 posted 11:01 pm on Jul 5, 2008 (gmt 0)

If I thought there was any chance at all that they set up a DNS system that could not only handle every Google link from 20 million customers, plus do a lookup for a hit in their own database, I'd spend 30 minutes reinstalling the thing I installed earlier today and run Wireshark at the same time. But I don't think there's any chance at all. What are the odds that they're doing this, based on everything we've seen from them lately?

This 219 message thread spans 8 pages: < < 219 ( 1 2 3 4 5 6 [7] 8 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved