homepage Welcome to WebmasterWorld Guest from 54.204.94.228
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

This 219 message thread spans 8 pages: < < 219 ( 1 2 [3] 4 5 6 7 8 > >     
Register Scolds AVG For Generating Fake Traffic As Link Malware
Webmasters Complain AVG Debilitating Traffic Analytics
Samizdata




msg:3674412
 8:52 pm on Jun 13, 2008 (gmt 0)

In an otherwise interesting article about AVG LinkScanner the author spectacularly misses the point that because it can easily be identified it is worse than useless as a security tool.

But he does tell malware infested drive-by download sites how to fool it.

[theregister.co.uk...]

...

 

appi2




msg:3680952
 7:48 pm on Jun 22, 2008 (gmt 0)

AVG paid version, updated yesterday.

If you Google any of my sites they will feed LinkScanner a dummy file and get AVG approval.

Nice to know you care more about your sats than your users. Now if your site is hacked an some exploit appears AVG users may get hit as well as non AVG users. Don't call that being one of the good guys.

I'll have a look at my stats tommorow and see if the user agent changed.

incrediBILL




msg:3680955
 8:00 pm on Jun 22, 2008 (gmt 0)

appears AVG users may get hit as well as non AVG users

You assume AVG's LinkScanner would detect it and stop it in the first place which I proved it doesn't always.

Roll them dice ;)

Samizdata




msg:3680966
 8:17 pm on Jun 22, 2008 (gmt 0)

AVG paid version, updated yesterday

AVG updates itself every day (or every four hours if you choose).

After your post I downloaded the latest free version, updated it, and did a test.

I recommend that you do likewise before demonstrating your lack of understanding.

Nice to know you care more about your sats than your users

I don't care about statistics at all.

I do care about my visitors, which is why I don't allow unwanted and useless robots.

Don't call that being one of the good guys

At least I don't sell snake-oil to suckers like AVG is doing.

...

Samizdata




msg:3680967
 8:23 pm on Jun 22, 2008 (gmt 0)

Let's keep it simple for any latecomers:

AVG LinkScanner is a security risk and endangers anyone who uses it.

It is so easy to fool that it makes AVG Technologies look incompetent.

AVG has also handed a database of their users' IPs to the "Bad Guys".

AVG's head of communications has been told as much.

If LinkScanner is not withdrawn or changed so much as to be unrecognisable and a completely different product then the case for negligence would seem overwhelming.

...

[edited by: Samizdata at 8:24 pm (utc) on June 22, 2008]

[edited by: incrediBILL at 9:42 pm (utc) on June 22, 2008]
[edit reason] removed obsolete comment in thread cleanup [/edit]

Receptional Andy




msg:3680970
 8:40 pm on Jun 22, 2008 (gmt 0)

Now if your site is hacked an some exploit appears AVG users may get hit as well as non AVG users. Don't call that being one of the good guys.

This touches on a key point, mentioned several times but worth repeating.

LinkScanner does not provide an added level of security. It only protects you in the unlikely and somewhat revealing scenario that AVG can flag a site as harmful, but be unable to protect you from harm in the event you click through to the site.

The release of LinkScanner appears to suggest that AVG is able to recognise certain types of malicious software, but is unable to protect you from it. So either you were already (and are currently) safe, or you were never safe at all.

Worse still, intrinsic and basic problems with the implementation of LinkScanner have actually exposed the user-base to an increased level of risk.

It seems to me like a losing situation for everyone involved. So IMHO, the best plan is for it to stop altogether ;)

appi2




msg:3680988
 9:00 pm on Jun 22, 2008 (gmt 0)

Right I must uninstall it and rely on google or maybe firefox/opera or maybe that PC treacle Norton. From a user point of veiw I like it. From a webmaster point of view. Not good.

Anyhoo

Checked the logs.
User agent is
"User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Used FF3

Did a site:www.example.com in google and then checked the logs. The ip for the hits were my IP, but I didn't visit the site.

Sorry not read the whole thread, if thats any help then carry on.

Samizdata




msg:3680993
 9:07 pm on Jun 22, 2008 (gmt 0)

Sorry not read the whole thread, if thats any help then carry on.

Appi2 that is indeed helpful.

What it (and other reports) tells us is that someone at AVG is trying to fix LinkScanner.

It also tells us that they have no idea what they are doing and are bound to fail.

But then we already knew that, and have told them so.

...

Samizdata




msg:3681021
 10:00 pm on Jun 22, 2008 (gmt 0)

Did a site:www.example.com in google and then checked the logs

There are many ways to fool LinkScanner, but those with a sense of humour seem to like this:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ;1813\)$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^User
RewriteRule /*$ [grisoft.com...] [L]

I have also seen it suggested that installing LinkScanner, setting your Google results to 100, doing a "site:example.com" for AVG's site and refreshing the page frequently is one way to fill a boring hour.

I would never do such a thing myself, though.

I am one of the "Good Guys".

...

[edited by: Samizdata at 10:10 pm (utc) on June 22, 2008]

incrediBILL




msg:3681030
 10:14 pm on Jun 22, 2008 (gmt 0)

refreshing the page frequently is one way to fill a boring hour.

Think automation.

Use the Windows Task Scheduler to do it every 5 minutes.

Not like I would do such a thing either. ;)

appi2




msg:3681071
 11:42 pm on Jun 22, 2008 (gmt 0)

Thought you might like to know;)

Both the User Agent I listed above and the 1813 one are in the file avgssff.dll

in C:\Program Files\AVG\AVG8\Firefox\Components

Samizdata




msg:3681093
 12:13 am on Jun 23, 2008 (gmt 0)

Not like I would do such a thing either.

I certainly didn't mean to encourage such behaviour, and anyone who does it from an IP traceable to them may find that it backfires if AVG complain they are being DoS'd with their own software.

My own "Good Guy" code of ethics demands that I deal with the LinkScanner problem in-house, which is why I serve it a small dummy file, but I would not criticize anyone who felt like passing the bandwidth cost back to AVG.

This whole thing is such a farce that it is hard to stay serious.

But I recognise that drive-by downloads are a serious problem.

LinkScanner is not a serious response to it, though.

...

incrediBILL




msg:3681118
 1:03 am on Jun 23, 2008 (gmt 0)

LinkScanner is not a serious response to it, though.

Sadly, neither are some of the other AV products as I'm slowly finding out.

I have a sample that is such a pure IFRAME 101 hack that anything should detect it and several are glossing right over it, so LinkScanner is in plenty of bad company.

Brett_Tabke




msg:3681119
 1:06 am on Jun 23, 2008 (gmt 0)

anyone mention that this link scanner will blow the 'first click free' process out of the water?

- The link scanner dls a link, and it's ip is recorded by the site
- he/she clicks the link.
- access denied

g1smd




msg:3681265
 7:53 am on Jun 23, 2008 (gmt 0)

How many reasons do we need to find before they admit it is a flawed product?

We must be into double-digits by now...

blend27




msg:3681280
 8:43 am on Jun 23, 2008 (gmt 0)

--- How many reasons --

Four Hundred and 3

:)

Romeo




msg:3681321
 9:58 am on Jun 23, 2008 (gmt 0)

User agent is
"User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

LOL! ROFL!
What a great choice. Highly professional, really. Thanks, that made my day!

I have seen some shady clients using similar UAs long time before and didn't like it, so this kind of UA is already in my long standing scraper block list.

Unbelievable.
R.

Samizdata




msg:3681346
 11:16 am on Jun 23, 2008 (gmt 0)

anyone mention that this link scanner will blow the 'first click free' process out of the water?

I believe you are the first Brett.

There are so many problems with LinkScanner that attempting to list them all (my first thought) would take more time than I have available. If someone wants to try, here are some references:

--

A Beginners Guide to AVG LinkScanner

WebmasterWorld member Umbra was the first to report a problem (in his case with mod_security) on 31 March, and WebmasterWorld member DanA was the first to link the problem and the "1813" user-agent to AVG on 28 April.

[webmasterworld.com...]

Much confusion followed, partly due to there being two versions of LinkScanner on the loose (the AVG one and the original Exploit Prevention Labs one) and also due to general disbelief that a respected security company like AVG could do something so crass.

The thread does contain much useful information and some excellent solutions.

There was a misleading suggestion that Trend Micro was involved and the thread was split.
[webmasterworld.com...]

After some fairly comprehensive testing I posted this disambiguation.
[webmasterworld.com...]

Meanwhile WebmasterWorld member Wlauzon had his site banned completely.
[webmasterworld.com...]

--

Others have been more concerned with the effect on analytics:

WebmasterWorld member Cyclob noticed a "strange traffic increase".
[webmasterworld.com...]

WebmasterWorld member dmje "had over 1700 hits all by the same user agent".
[webmasterworld.com...]

--

Some reports have been inconclusive and LinkScanner may not be involved:

WebmasterWorld member zahirshah said that his "bounce rate dramatically went to 50/60%".
[webmasterworld.com...]

WebmasterWorld member g1smd reported a problem with the base tag.
[webmasterworld.com...]

--

Readers can be forgiven for finding all this stuff confusing, but the information on WebmasterWorld is far better than anything else on the web - including The Register and AVG's own forum.

My personal thanks to all who contributed.

...

incrediBILL




msg:3681639
 4:52 pm on Jun 23, 2008 (gmt 0)

According to a new article on The Register the AVG LinkScanner doesn't click sponsored links.

[theregister.co.uk...]

But in scanning sponsored sites, AVG is careful to bypass the Google mechanism that records paid clicks. Rather than use Google's hyperlink, it uses the site's raw URL. "We parse out the target and go straight there, skipping any Google click counter," says Pat Bitton, head of communications at AVG, a Czech company with regional offices in the US and the UK.

Unless someone can prove otherwise, the ability of LinkScanner to rack up AdWords charges has been written off as urban LinkScanner legend.

Samizdata




msg:3681663
 5:10 pm on Jun 23, 2008 (gmt 0)

I posted that link further up the thread three days ago.

Our friends in the Adwords forum had already debunked the legend over a month previously:

[webmasterworld.com...]

The interesting part of The Register's article for me was this:

His chief concern is security, and he doesn't want webmasters or malware writers gaming his scanner

Fair enough, but as I pointed out at the time everybody already had been gaming it for ages.

...

incrediBILL




msg:3681721
 6:27 pm on Jun 23, 2008 (gmt 0)

Fair enough, but as I pointed out at the time everybody already had been gaming it for ages.

Also, as I pointed out the other day, it blatantly misses some of the simplest iFrame malware code so it's not even catching the simple stuff, let alone people gaming it.

g1smd




msg:3682274
 11:17 am on Jun 24, 2008 (gmt 0)

... AVG has assembled a consultative group of webmasters and analytics folks, including Adam Beale, who originally reported this issue to The Register, to work through this issue ...

Does that include anyone from here perchance?

Romeo




msg:3682309
 12:10 pm on Jun 24, 2008 (gmt 0)

... to work through this issue ...

Ah, yes, they work. They renamed the User-Agent.

anyone from here perchance?

Well, most likely not. From this UA rename we can see that they already have enough professional experts on board, so they eventually don't need us ...

Are we having fun yet?

Samizdata




msg:3682320
 12:36 pm on Jun 24, 2008 (gmt 0)

Does that include anyone from here perchance?

As one of the "Good Guys" it is not my practise to make private emails public, and I am sure such a thing would be against the rules of WebmasterWorld anyway - they are the "Good Guys" too.

But it is no secret that I had an exchange of emails with Pat Bitton of AVG as soon as s/he appealed for help after the first article in The Register, and I have previously mentioned that s/he spurned my offer of assistance after reading this thread.

Pat seemed to consider us the "Bad Guys", and that is very sad, because the assembled talents of WebmasterWorld are the very people AVG needs on their side - we are their natural allies, and folks here have been fighting the "Bad Guys" of the web a lot longer than AVG has.

So now we find ourselves at war - AVG does a bit of tinkering with LinkScanner and webmasters take a couple of minutes to find a new workaround, and it looks like this process will continue for the foreseeable future.

Anyone reading the many threads about LinkScanner will notice the animosity. I take my share of the responsibility, as I have been a sharp and frequent critic, but I have no interest in visiting grief on AVG - like others here I have recommended their anti-virus product to my friends for several years. I just can't help laughing at their silly antics.

But AVG must eventually understand that their public statements dismissing webmasters' concerns did them no favours at all. They already had an enemy in the "Bad Guys", and they seemed to go out of their way to create a new enemy with their fighting talk about "breaking eggs", even though it was abundantly clear that their product was flawed and causing serious problems.

The nature of WebmasterWorld means that ways to fool LinkScanner will always be found and will always be made public, so even the most dim-witted "Bad Guys" will know how to defeat it within a few hours of any new release. Likewise the folks at AVG will immediately redouble their efforts. All this waste will continue until AVG realise that they are fighting a battle they cannot win.

And there's the rub. Pat Bitton told me that s/he would post on WebmasterWorld as soon as AVG had a solution in place. But AVG's current approach will never find a solution, and will only antagonise webmasters even further.

Some WebmasterWorld members - jdMorgan in particular - have spelled out exactly what AVG needs to do to fix their broken product. All AVG has to to is hire somebody who knows what they are doing.

...

g1smd




msg:3682324
 12:45 pm on Jun 24, 2008 (gmt 0)

Am I allowed to mention that Pat Bitton is posting to Sphinn now?

Yeah, really!

Look for topic number 54536 or put "AVG" in their search box.

Staffa




msg:3682574
 5:23 pm on Jun 24, 2008 (gmt 0)

They are tinkering, sorry working through this issue ...

First I get a visit from the new and improved UA :

User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

calling a HEAD and GET within the same second. I have since long 'User-Agent: whatever' banned so they got nothing, though visitor came through all the same.

Four hours later it was back to straight Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

Keep up the good work guys, eventually you'll beat the world record for largest omelet.

Samizdata




msg:3682636
 6:32 pm on Jun 24, 2008 (gmt 0)

An updated free version was released yesterday (23 June).

LinkScanner currently uses these user-agents:

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)

"In order to detect the really tricky - and by association, the most important - malicious content, we need to look just like a browser driven by a human being" (Roger Thompson)

...

incrediBILL




msg:3682710
 8:15 pm on Jun 24, 2008 (gmt 0)

we need to look just like a browser driven by a human being

Browsers driven by human beings don't do HEAD/GET and they don't use ";1813" or a "User-Agent:" prefix and they don't do a few other flawed things LinkScanner does.

Getting the user agent to properly mimic the actual browser being used is so simple anyone could do it so why can't they?

dstiles




msg:3682791
 10:34 pm on Jun 24, 2008 (gmt 0)

The "user-agent:" prefix is not exclusively AVG. I've been trapping it, as I said earlier, for years. I suggest that anything not matching the exact prefixed string (or at lease the prefix with either "SV1)" or "1813") be treated as a bad bot.

Romeo




msg:3683030
 11:59 am on Jun 25, 2008 (gmt 0)

Well, it will perhaps only take a few more days until their experts will find out a UA string that looks like a real browser UA string that we webmasters finally couldn't distinguish from real users.
They would be done then. Mission accomplished.

And we webmasters will be left with the fallout of bloated and abusive traffic volume and worthless usage statistics.

If they just continue with their intrusive rudeness, 'we' (webmasters) are simply lost.

Thank you, AVG, thank you.

They may feel like a 'winner' in the short run, but in the long run ... perhaps they may loose some market share due to bad press.

R.

Samizdata




msg:3683127
 2:45 pm on Jun 25, 2008 (gmt 0)

UA string that looks like a real browser UA string

A point that has been rarely made plainly in this interminable fiasco is that LinkScanner lies.

LinkScanner pretends - albeit laughably - to be something it is not, a human visitor

AVG seem to think it is reasonable to try to deceive webmasters in this way.

I would say such dishonesty deserves the contempt it is getting.

...

[edited by: Samizdata at 3:08 pm (utc) on June 25, 2008]

mlduclos




msg:3683410
 7:29 pm on Jun 25, 2008 (gmt 0)

Hello

My server is being overloaded by this stupid new "feature" for AVG. I think people should continue to notify AVG about this problem, so they can fix it.

My DB is causing a huge increase of server load and I got 2 Ddos already
. Can someone please send me a .htaccess code to redirect user agent

"User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

to a static html page?
Thanks

This 219 message thread spans 8 pages: < < 219 ( 1 2 [3] 4 5 6 7 8 > >
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved