homepage Welcome to WebmasterWorld Guest from 54.196.162.238
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 173 message thread spans 6 pages: < < 173 ( 1 2 3 [4] 5 6 > >     
AVG Toolbar Glitch May Be Causing Visitor Loss
User Agent Flaw Suspected
Umbra




msg:3615362
 2:36 pm on Mar 31, 2008 (gmt 0)

Seeing a rash of hits with an oddly formed user agent:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)
No referer

mod_security always throws an error for this one. Hits come from various IPs with no consistent pattern, seem to be residential IPs. Any idea what it is?

 

jdMorgan




msg:3647560
 11:37 pm on May 11, 2008 (gmt 0)

As suggested in several posts earlier in this thread, returning a very small valid html page to the AVG Linkscanner client is a much safer way to conserve bandwidth without risking traffic or revenue loss. In .htaccess:

RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^a-very-small-page\.html$ /a-very-small-page.html [L]

Jim

blend27




msg:3647566
 12:09 am on May 12, 2008 (gmt 0)

and as far as AdWords goes on the same page, 8 out of 10 Advertisers had a big grey question mark next to the Ad, including EBAY, HSN, JTV and other big PLAYAS in a given niche, so there goes CTR...

smallcompany




msg:3647572
 12:47 am on May 12, 2008 (gmt 0)

Funny, I haven't seen anything except green checkmarks. I just queried ebay and hsn and got all greens. Specific to a landing page, right?

incrediBILL




msg:3647577
 1:02 am on May 12, 2008 (gmt 0)

Funny, I haven't seen anything except green checkmarks.

The pages would most likely have to contain a virus injector or phishing code to cause the toolbar to signal an alert.

I have a bunch of URLs that *should* set it off but I can't post them here, but I'll post the results when I get around to testing them.

Samizdata




msg:3647581
 1:16 am on May 12, 2008 (gmt 0)

403 puts a BIG green check mark to the right of the listing in SERP

My tests consistently say 403 = grey question mark (and no approval from AVG).

My 403 is triggered by an NT version-checking routine that expects a space after the semi-colon.

Apache responded this way to 29 identical requests in 9 seconds.

Meanwhile I confirm that I was not pressured into allowing access to this user-agent.

Grisoft simply made me an offer I couldn't refuse...

incrediBILL




msg:3649048
 5:44 pm on May 13, 2008 (gmt 0)

Been watching this toolbar in my log files and it's added 7,053 to my page views so far this month and it's escalating daily.

Here's the last few days of AVG toolbar traffic in order: 592, 662, 720, 905, anyone else seeing this growing trend?

wilderness




msg:3649078
 6:20 pm on May 13, 2008 (gmt 0)

Wonder if it's possible this increase has anything to do with the recent release of SP3 update for XP?

smallcompany




msg:3649164
 7:45 pm on May 13, 2008 (gmt 0)

Isn't increase related directly to the fact that people are upgrading/installing new AVG 8.0?
Previous AVG was not doing this link check.

All this with the assumption that it is AVG, both 1813 and SV1 UAs.

Samizdata




msg:3649209
 8:22 pm on May 13, 2008 (gmt 0)

anyone else seeing this growing trend?

I am - though no AVG installations that I upgrade myself are getting the toolbar, obviously.

possible this increase has anything to do with the recent release of SP3 update for XP?

In my (albeit uneducated) opinion it is entirely unrelated to SP3.

All this with the assumption that it is AVG, both 1813 and SV1 UAs

I do not relate the term SV1 to the AVG toolbar user-agent at all.

Can we expect similar from Symantec, MacAfee et al, or do we already have them?

I asked this question earlier - can anyone answer it?

--

I also notice from this thread [webmasterworld.com] that AVG is having fun banning innocent sites on shared IPs.

[edited by: encyclo at 10:42 pm (utc) on May 13, 2008]
[edit reason] fixed link [/edit]

dstiles




msg:3649356
 2:24 am on May 14, 2008 (gmt 0)

Further to an early mention of the UA...

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)

The pattern I'm seeing for this is extremely similar to the 1813 case - single IPs, missing ACCEPT and no referer. It also almost invariably has some a querystring tracer missing, which in this case is indicative of it coming from a search engine and most certainly not from elsewhere within the site.

It seems possible to me it could either be an earlier version of an AVG signature, possibly from another OS, or perhaps from another AV/Firewall company.

smallcompany




msg:3649393
 3:35 am on May 14, 2008 (gmt 0)

On my sites, Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) behaves absolutely the same as 1813.

I actually asked about this one back in March. That is when I first noticed its strange behavior, but wasn’t aware of its background. That is why I created a post in Browsers section:

[webmasterworld.com...]

Vamm




msg:3649519
 8:09 am on May 14, 2008 (gmt 0)

The increase is not related to SP3.

The reason is
AVG 7.5 (with NO toolbar) is currently displays banners stating version 7.5 is end-of-life. Although deadline is declared 31/12/2008, people already starting to upgrade (to AVG 8.0 with toolbar), because of the banner.

Samizdata




msg:3649680
 1:29 pm on May 14, 2008 (gmt 0)

Jim was right (as usual).

This user-agent does not come from the "Security Toolbar" (which is an optional, if pre-checked, install) but from the AVG LinkScanner component (which is installed by default).

I upgraded another three AVG installations today, each without the toolbar, and all had their search results interfered with by the AVG internet police, wasting bandwidth and leaving the tell-tale footprint Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813) (which as we have seen is easily fooled, making it useless or worse).

Once again, sites that gave it a 403 were "greylisted" and given an effective thumbs-down.

Grisoft, you just jumped the shark.

mindaugas13




msg:3650143
 8:57 pm on May 14, 2008 (gmt 0)

I too installed AVG Free 8.0 and verified that it visits sites with this user agent:

mozilla/4.0 (compatible; msie 6.0; windows nt 5.1;1813)

This is actually done by the LinkScanner feature of AVG 8.0. When I installed AVG, I specifically did not install the Security Toolbar (option during custom installation). However the Link Scanner is part of the main AVG program. It can be disabled, but then the AVG icon appears as a red exclamation mark.

johnnie




msg:3650376
 12:36 am on May 15, 2008 (gmt 0)

By the way, in IE the search result-scanner can be easily disabled through internet options - > programs -> add-ons. Just disable the safe search. Seriously, who makes up this crap...

smallcompany




msg:3650460
 2:19 am on May 15, 2008 (gmt 0)

Since AVG is on the famous highway towards becoming bloatware, we can only hope for a massive shift towards avira.

AVG did enough to promote itself. I did grab Avira recently as a part of my AV testing, but how many of ordinary people care abut this? None.

By the way, in IE the search result-scanner can be easily disabled through internet options - > programs -> add-ons. Just disable the safe search. Seriously, who makes up this crap...

Continuing from the above, an average user never goes to any of the options under menu.

It is just that all of these companies are trying to stay in the game by inventing something “new”, something that (like) adds an extra protection layer.
Most of that turns to be a marketing move, and nothing else.

That is why I took long time good AV off my machine and had it (my PC) run like never before.

Back to the topic… It is about all these companies, including Grisoft, to ensure they don’t interfere with other people’s business, in any meaning. If they want to do something like this, they better do it transparently.

spotter




msg:3651076
 5:43 pm on May 15, 2008 (gmt 0)

I’ve been watching this thread with interest and finally had to chip in. Check out FAQ 1338 on the Grisoft site and it gives you a command line call to install AVG8 without the Linkscanner. You can indicate that the FAQ was helpful, so if enough of us do it they may get the message, even if you have no intention of using AVG.

Apparently Grisoft purchased Exploit Prevention Labs to acquire this useless piece of software. Guess it was cheap.

I’d been noticing hits from the product in my logs for some months but could not figure what they were until it was incorporated into AVG. I have an external js function with a variable passed in as a parameter and called with +<varname>+. This trips it up and it returns a 404 with the code snippet rather than the filename. The site still shows as approved in the SERP.

System
redhat



msg:3651562
 3:21 am on May 16, 2008 (gmt 0)

several messages were cut out to new thread by incredibill. New thread at: search_engine_spiders/3651560.htm [webmasterworld.com]
9:45 pm on May 15, 2008 <small>(PST -8)</small>

[edited by: jatar_k at 12:12 pm (utc) on May 16, 2008]

smallcompany




msg:3657692
 7:27 pm on May 23, 2008 (gmt 0)

...but this thread is still about 1813, right?

incrediBILL




msg:3657701
 7:40 pm on May 23, 2008 (gmt 0)

Yes, all about ";1813" ;)

blend27




msg:3658183
 3:15 pm on May 24, 2008 (gmt 0)

I wonder if the number has anything to do with "sic itur ad astra"

Umbra




msg:3658268
 6:22 pm on May 24, 2008 (gmt 0)

Once again, sites that gave it a 403 were "greylisted" and given an effective thumbs-down.

How does AVG respond to an error 500 response?

RSweetnam




msg:3662024
 4:45 pm on May 29, 2008 (gmt 0)

I've been seeing similar behaviour on my site although the really strange this is that all the IP address where I see the useragent from are all located in my home country and from different ISP's.

[edited by: incrediBILL at 5:23 pm (utc) on May 29, 2008]
[edit reason] URL removed, see TOS #13 & #25 [/edit]

jdMorgan




msg:3662091
 5:31 pm on May 29, 2008 (gmt 0)

I've been using the "deliver a small, simple page with one link" method since the day we figured out this was AVG LinkScanner. I've had no problems, but wasted bandwidth is way down. :)

The link on my short/sweet page just links to my home page -- It doesn't seem to need to be any specific link.

Jim

superclown2




msg:3664155
 11:38 am on Jun 1, 2008 (gmt 0)

Jim, please excuse what may sound like a very uneducated query but I confess that mod-rewrite scares the h*ll out of me, so when you write:

RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^a-very-small-page\.html$ /a-very-small-page.html [L]

I appreciate that the second 'a-very-small-page.html' refers to the URL of a, well, very small page, but what do I change 'a-very-small-page\.html$' to, please?

Samizdata




msg:3664217
 2:39 pm on Jun 1, 2008 (gmt 0)

It should be the same filename, as this prevents creating an infinite loop.

The rule says "if the request is NOT for the substitute file, serve the substitute file".

The dot before the extension needs to be escaped in the first part of the rule.

superclown2




msg:3664400
 8:46 pm on Jun 1, 2008 (gmt 0)

Thanks, I'm very grateful but I can't get it to work. I created a file called block.html and this is my .htaccess file:
RemoveHandler .html .htm
AddType application/x-httpd-php .php .htm .html
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^block\.html$ /block.html [L]

Please, any suggestions about what I'm doing wrong?

Samizdata




msg:3664427
 9:56 pm on Jun 1, 2008 (gmt 0)

Much depends on how your server is set up and what your other directives are supposed to do, but I would say the likely cause of your problem is the RemoveHandler directive.

Why not try naming your substitute file block.php instead?

# Set options (may be required)
Options +FollowSymlinks

# Turn on mod_rewrite
RewriteEngine On

# Deal with idiotic prefetch
RewriteCond %{HTTP_USER_AGENT} ;1813\)$
RewriteRule !^block\.php$ /block.php [L]

superclown2




msg:3664630
 7:45 am on Jun 2, 2008 (gmt 0)

Ah! This is the reason, the user_agent is different:
"GET /my-web-page.html HTTP/1.1" 200 32961 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

What is user agent SV1? Should I be rewriting that instead or will that kill legitimate traffic?

I'm amazed that this company seems perfectly happy to screw up the whole world's web stats, this could develop into something very interesting.

superclown2




msg:3664736
 11:38 am on Jun 2, 2008 (gmt 0)

"What is user agent SV1? Should I be rewriting that instead or will that kill legitimate traffic?"

Sorry I really need to learn to read a thread properly without asking questions that have already been answered.

Leaving aside the mess in all our stats and the inevitable trouble that this will cause Grisoft when all those angry webmasters realise who is responsible for it I really can't see the need for this pre-loading when Google already label dangerous sites, and since I installed the toolbar on one of my high-spec computers with 16 meg broadband the load time for Internet Explorer has increased to about 15-20 seconds so I for one have removed it and put Norton back on. The sooner they drop this pile of xyz the better for their business as well as ours.

rise2it




msg:3666362
 4:44 am on Jun 4, 2008 (gmt 0)

I skipped about 40 posts, so excuse me if this got answered earlier...but it's probably NOT only the toolbar.

The main AVG screen has a settings block called 'link scanner' which can be disabled, but is enabled by default. This, in turn, throws up extra icons on Google pages in both IE and Firefox.

This means it's 'on' for every single person who has installed the software.

I assume if you come up in the search results page, it's doing a hit of your webpage (and every other page coming up in the search results), which is where this is coming from - so the end user may NEVER even click to go to your site, and you're still going to be showing a referral from this stuff, even though you never got a real visitor.

For instance, if you have Google set to show 10 results by default, AVG is going to pull all 10....if your search page is set for 20, 50, or 100 results then AVG is going to go pull ALL of those pages, and EVERY ONE of those sites are going to show this referrer.

*** That would explain the extra hits you are showing ***

Of course, that means it gets WORSE, because...

Now, if you somehow block AVG, then it's going to show you as being potentially 'bad' in the search engine results page, causing you to lose potential visitors.

I'm afraid to go look at any of MY referrer logs now...grrrrrrr

This 173 message thread spans 6 pages: < < 173 ( 1 2 3 [4] 5 6 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved