AVG Toolbar Glitch May Be Causing Visitor Loss

User Agent Flaw Suspected

[Umbra]

Seeing a rash of hits with an oddly formed user agent:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)
No referer

mod_security always throws an error for this one. Hits come from various IPs with no consistent pattern, seem to be residential IPs. Any idea what it is?

[jdMorgan]

As suggested in several posts earlier in this thread, returning a very small valid html page to the AVG Linkscanner client is a much safer way to conserve bandwidth without risking traffic or revenue loss. In .htaccess:

RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^a-very-small-page\.html$ /a-very-small-page.html [L]

Jim

[blend27]

and as far as AdWords goes on the same page, 8 out of 10 Advertisers had a big grey question mark next to the Ad, including EBAY, HSN, JTV and other big PLAYAS in a given niche, so there goes CTR...

[smallcompany]

Funny, I haven't seen anything except green checkmarks. I just queried ebay and hsn and got all greens. Specific to a landing page, right?

[incrediBILL]

Funny, I haven't seen anything except green checkmarks.

The pages would most likely have to contain a virus injector or phishing code to cause the toolbar to signal an alert.

I have a bunch of URLs that *should* set it off but I can't post them here, but I'll post the results when I get around to testing them.

[Samizdata]

403 puts a BIG green check mark to the right of the listing in SERP

My tests consistently say 403 = grey question mark (and no approval from AVG).

My 403 is triggered by an NT version-checking routine that expects a space after the semi-colon.

Apache responded this way to 29 identical requests in 9 seconds.

Meanwhile I confirm that I was not pressured into allowing access to this user-agent.

Grisoft simply made me an offer I couldn't refuse...

[incrediBILL]

Been watching this toolbar in my log files and it's added 7,053 to my page views so far this month and it's escalating daily.

Here's the last few days of AVG toolbar traffic in order: 592, 662, 720, 905, anyone else seeing this growing trend?

[wilderness]

Wonder if it's possible this increase has anything to do with the recent release of SP3 update for XP?

[smallcompany]

Isn't increase related directly to the fact that people are upgrading/installing new AVG 8.0?
Previous AVG was not doing this link check.

All this with the assumption that it is AVG, both 1813 and SV1 UAs.

[Samizdata]

anyone else seeing this growing trend?

I am - though no AVG installations that I upgrade myself are getting the toolbar, obviously.

possible this increase has anything to do with the recent release of SP3 update for XP?

In my (albeit uneducated) opinion it is entirely unrelated to SP3.

All this with the assumption that it is AVG, both 1813 and SV1 UAs

I do not relate the term SV1 to the AVG toolbar user-agent at all.

Can we expect similar from Symantec, MacAfee et al, or do we already have them?

I asked this question earlier - can anyone answer it?

--

I also notice from this thread [webmasterworld.com] that AVG is having fun banning innocent sites on shared IPs.

[edited by: encyclo at 10:42 pm (utc) on May 13, 2008]
[edit reason] fixed link [/edit]

[dstiles]

Further to an early mention of the UA...

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

The pattern I'm seeing for this is extremely similar to the 1813 case - single IPs, missing ACCEPT and no referer. It also almost invariably has some a querystring tracer missing, which in this case is indicative of it coming from a search engine and most certainly not from elsewhere within the site.

It seems possible to me it could either be an earlier version of an AVG signature, possibly from another OS, or perhaps from another AV/Firewall company.

[smallcompany]

On my sites, Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) behaves absolutely the same as 1813.

I actually asked about this one back in March. That is when I first noticed its strange behavior, but wasn’t aware of its background. That is why I created a post in Browsers section:

[webmasterworld.com...]

[Vamm]

The increase is not related to SP3.

The reason is
AVG 7.5 (with NO toolbar) is currently displays banners stating version 7.5 is end-of-life. Although deadline is declared 31/12/2008, people already starting to upgrade (to AVG 8.0 with toolbar), because of the banner.

[Samizdata]

Jim was right (as usual).

This user-agent does not come from the "Security Toolbar" (which is an optional, if pre-checked, install) but from the AVG LinkScanner component (which is installed by default).

I upgraded another three AVG installations today, each without the toolbar, and all had their search results interfered with by the AVG internet police, wasting bandwidth and leaving the tell-tale footprint Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813) (which as we have seen is easily fooled, making it useless or worse).

Once again, sites that gave it a 403 were "greylisted" and given an effective thumbs-down.

Grisoft, you just jumped the shark.

[mindaugas13]

I too installed AVG Free 8.0 and verified that it visits sites with this user agent:

mozilla/4.0 (compatible; msie 6.0; windows nt 5.1;1813)

This is actually done by the LinkScanner feature of AVG 8.0. When I installed AVG, I specifically did not install the Security Toolbar (option during custom installation). However the Link Scanner is part of the main AVG program. It can be disabled, but then the AVG icon appears as a red exclamation mark.

[johnnie]

By the way, in IE the search result-scanner can be easily disabled through internet options - > programs -> add-ons. Just disable the safe search. Seriously, who makes up this crap...

[smallcompany]

Since AVG is on the famous highway towards becoming bloatware, we can only hope for a massive shift towards avira.

AVG did enough to promote itself. I did grab Avira recently as a part of my AV testing, but how many of ordinary people care abut this? None.

By the way, in IE the search result-scanner can be easily disabled through internet options - > programs -> add-ons. Just disable the safe search. Seriously, who makes up this crap...

Continuing from the above, an average user never goes to any of the options under menu.

It is just that all of these companies are trying to stay in the game by inventing something “new”, something that (like) adds an extra protection layer.
Most of that turns to be a marketing move, and nothing else.

That is why I took long time good AV off my machine and had it (my PC) run like never before.

Back to the topic… It is about all these companies, including Grisoft, to ensure they don’t interfere with other people’s business, in any meaning. If they want to do something like this, they better do it transparently.

[spotter]

I’ve been watching this thread with interest and finally had to chip in. Check out FAQ 1338 on the Grisoft site and it gives you a command line call to install AVG8 without the Linkscanner. You can indicate that the FAQ was helpful, so if enough of us do it they may get the message, even if you have no intention of using AVG.

Apparently Grisoft purchased Exploit Prevention Labs to acquire this useless piece of software. Guess it was cheap.

I’d been noticing hits from the product in my logs for some months but could not figure what they were until it was incorporated into AVG. I have an external js function with a variable passed in as a parameter and called with +<varname>+. This trips it up and it returns a 404 with the code snippet rather than the filename. The site still shows as approved in the SERP.

[System]

several messages were cut out to new thread by incredibill. New thread at: search_engine_spiders/3651560.htm [webmasterworld.com]
9:45 pm on May 15, 2008 <small>(PST -8)</small>

[edited by: jatar_k at 12:12 pm (utc) on May 16, 2008]

[smallcompany]

...but this thread is still about 1813, right?

[incrediBILL]

Yes, all about ";1813" ;)

[blend27]

I wonder if the number has anything to do with "sic itur ad astra"

[Umbra]

Once again, sites that gave it a 403 were "greylisted" and given an effective thumbs-down.

How does AVG respond to an error 500 response?

[RSweetnam]

I've been seeing similar behaviour on my site although the really strange this is that all the IP address where I see the useragent from are all located in my home country and from different ISP's.

[edited by: incrediBILL at 5:23 pm (utc) on May 29, 2008]
[edit reason] URL removed, see TOS #13 & #25 [/edit]

[jdMorgan]

I've been using the "deliver a small, simple page with one link" method since the day we figured out this was AVG LinkScanner. I've had no problems, but wasted bandwidth is way down. :)

The link on my short/sweet page just links to my home page -- It doesn't seem to need to be any specific link.

Jim

[superclown2]

Jim, please excuse what may sound like a very uneducated query but I confess that mod-rewrite scares the h*ll out of me, so when you write:

RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^a-very-small-page\.html$ /a-very-small-page.html [L]

I appreciate that the second 'a-very-small-page.html' refers to the URL of a, well, very small page, but what do I change 'a-very-small-page\.html$' to, please?

[Samizdata]

It should be the same filename, as this prevents creating an infinite loop.

The rule says "if the request is NOT for the substitute file, serve the substitute file".

The dot before the extension needs to be escaped in the first part of the rule.

[superclown2]

Thanks, I'm very grateful but I can't get it to work. I created a file called block.html and this is my .htaccess file:
RemoveHandler .html .htm
AddType application/x-httpd-php .php .htm .html
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^block\.html$ /block.html [L]

Please, any suggestions about what I'm doing wrong?

[Samizdata]

Much depends on how your server is set up and what your other directives are supposed to do, but I would say the likely cause of your problem is the RemoveHandler directive.

Why not try naming your substitute file block.php instead?

# Set options (may be required)
Options +FollowSymlinks

# Turn on mod_rewrite
RewriteEngine On

# Deal with idiotic prefetch
RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^block\.php$ /block.php [L]

[superclown2]

Ah! This is the reason, the user_agent is different:
"GET /my-web-page.html HTTP/1.1" 200 32961 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

What is user agent SV1? Should I be rewriting that instead or will that kill legitimate traffic?

I'm amazed that this company seems perfectly happy to screw up the whole world's web stats, this could develop into something very interesting.

[superclown2]

"What is user agent SV1? Should I be rewriting that instead or will that kill legitimate traffic?"

Sorry I really need to learn to read a thread properly without asking questions that have already been answered.

Leaving aside the mess in all our stats and the inevitable trouble that this will cause Grisoft when all those angry webmasters realise who is responsible for it I really can't see the need for this pre-loading when Google already label dangerous sites, and since I installed the toolbar on one of my high-spec computers with 16 meg broadband the load time for Internet Explorer has increased to about 15-20 seconds so I for one have removed it and put Norton back on. The sooner they drop this pile of xyz the better for their business as well as ours.