Logs Show Surge, but Not Human?

[rogerd]

On one site I work with, I've seen traffic go from 10K visits/day to 40K. The additional traffic looks human at first glance - it is captured by Google Analytics, It comes from diverse consumer IPs in the US and Europe (but not Asia), and the bounce rate is high but one out of ten visits or so loads another page.

On the non-human side, we have all of the traffic coming with no referrer, and it is all focused on a few pages that are hardly viral linkbait and would get one or two views on a good day. It's all IE (spread among 6 - 9), and a range of screen resolutions that look unusually aged (e.g., 1024x768).

Anecdotally, I've heard of a few other sites seeing this kind of traffic, but nobody knows what the purpose might be. It's not scraping, as it's the same pages that get hit. It's not intense enough to be an attack to take the site down, nor is the site likely to be the target of miscreants.

The level of traffic has gone up and down, but it's still happening.

Are any of your sites seeing this, and do you have any theories?

Any thoughts on screening this out of Analytics? It totally blows up time period comparisons.

[Archbob]

Now I have traffic that triggers adsense(as well as my other ads) but not analytics. WTH?

[Mario155]

These bots are now causing my site to crash almost daily. When it crashes it stays down for almost an hour, depending on how long it takes for my web hosting company to reboot the server, and then they have to reset a bunch of other stuff.

I have decided to give up on my website even though it was making good money at one point. However over the last few months the bots have taken over, and revenue has dropped by a large amount. I guess these bots have accomplished their mission, I quit.

[b0rg]

I've seen some interesting visitors to the blog post about it on my site, including some big ISPs and the Dept of Defense. So maybe some others are starting to take notice.

interesting. maybe it IS the dept of defense testing a covert botnet attack - and now looking to see what anyone out there was able to unearth :)

random targets, no identifiable purpose? sounds like test run to me

[Edge]

interesting. maybe it IS the dept of defense testing a covert botnet attack

I just looked in my back end and this botnet is still hitting my site. Botnet out of control...

[hottrout]

Just a little update, wondering if anyone else has noticed. This phantom traffic has stopped over the weekend. I have seen my direct traffic sources drop dramatically and at the same time my average visit duration has jumped up and my bounce rate has dropped.

Has anyone else noticed this? Maybe I can start to trust my google stats again after six months of nonsense.

[aboshakeeb]

Guys ! , what now .. any news ?
My direct traffic started at 26 Jan 2012 and keeps coming , it hits 2 internal pages and the homepage with over 20k / day.

any suggestions !

[teslacoilwhalesperm]

my site had 10k not set visitors this month, but the operating system said they were on Linux in GA. (10k from linux?)

I also was running a Display network only campaign on adwords(2MM impressions), and thinking this could be loss of attribution from my display ads on aggregator/3rd party sites, but the Linux not set at 10K visitors seems fishy to me.

[aboshakeeb]

Huge increase today , totally from Europe , most hits seems from Germany , used to be from the USA .

[Edge]

Ok, looks the botnet gave up on my website about nine days ago.

Least i know how to handle it next time we meet...

[hottrout]

I have not had traffic from bots for approx three months now. I just wanted to update this thread for the other people involved. My google stats have returned to normal and my bouncrate and time on site are now a lot better.

[wa desert rat]

My site is a PHPBB3 forum with about 300 visits a day, 5 page reads per visit. Yesterday Analytics reported a sudden surge of 48 visits - all of them in Palo Alto, California - which did not show up in the "Who is Online" listings. The "visitors" mostly cruised the same pages (members lists, index) and went up and down from 30 to 48 (all according to Analytics) while the forum indicated nothing at all more than the usual 10 to 18 users on line. I installed Piwik this morning in order to capture IP addresses and log visitors. No sign of slowdown or attack. Just weird.

[wa desert rat]

Tonight, between 18:10:59 and 18:18:51 local time (PDT) and again between 18:35:28 and 18:38:01 my site had another series of "active" users that seem suspicious. Two interesting points are that all but one was from "Trendmicro" (the oddball was from "Wtp-g3-maya10" but was from the same /8 ip subnet). All addresses are APNIC and issued to Japan. The log indicates that at least there were two incidents of two connections from two separate IP addresses simultaneously. All but one of these read the index page. I cannot tell whether there were any click-throughs.

Interesting, I thought.

[brokaddr]

Interesting topic, I posted this before I seen this thread: [webmasterworld.com...]

Related?

My visits appear organic. Some have referrers (which match the landing page), some don't. All different locations/ISP.
Most are IE6-9 and iPhone.

As of late, conversion has sunk to a virtual decade high, yet hits (or the appearance of hits) are average.

[modelman]

This thread is the best I've seen on this topic. My bounce rate has been hammered and it has most certainly harmed my organic traffic. I don't use adwords or adsense, or any other ads. This attack has cost me 3/4 of my normal traffic since January.


I look at 100% bounce rate along with 00.00 duration in G.A. to find the offenders, and then isolate them by small towns or normally low traffic cities, or other indicators listed below.

The attack seems to have changed in recent months. My most recent strange traffic data shows these things in common:

* 100% direct traffic
* 100% first time visitors
* Pages/visit = 1
* Average visit duration = 00:00:00
* Browser "Mozilla Compatible Agent" version 5
* 24 bit screen colors/ Screen resolution 1024x768
* 100% Java disabled
* 100% Windows OS
* 100% Flash enabled (version 11.4 r402)
* Network Domain = msn.com
* keyword = "not set"

The target pages are spread out over hundreds of pages, with the most popular ones that normally have the lowest bounce rates hit most often.

The most interesting things that are different from the previously posted reports is that my home page is never hit, and that it is all domestic American traffic, even though my customers are worldwide. All my foreign traffic looks normal. The version of Flash being used is also new to this thread.

I can find other 100% bounce rate/00.00 duration traffic which has Network/Domain = "unknown.unknown" or with Flash version set to 11.4 r402, but the most reliable predictor is if it comes from msn.com with a Mozilla compatible agent.

I'm exploring ways to survive without Google search traffic, but as far as I can tell, this mystery has not been solved, it has not gone away, and I'm convinced that there are many, many more businesses that don't even know they are being adversely affected by it.

My first thought was that it was a competitor who wanted my search terms. I've gone through many other theories similar to those listed in this thread, but my latest thought is that it might be from the websites that furnish web stats for a domain name, because it started getting bad when I started visiting these sites and entering my domain name. At the same time, I renewed my SSL certificate (which was expired for over a week), so that might also be related, although I don't see how it could be. It's far too isolated to particular locations, although it is spreading and diversifying quite a bit in the past few weeks.

I've fended off many kinds of attacks, but this one has me stumped. I don't feel so bad seeing how everyone else seems to be mystified, but I would like to solve this by figuring out exactly how it is being done. I don't accept defeat very easily. Any help is greatly appreciated.

[brokaddr]

Network Domain = msn.com

What do you mean by this? Everyone that's bouncing off your pages is on the MSN network?

My visitors are from all over the place and on different, legit ISPs. Some have referrers/keywords that a real visitor would be using to find my sites, some do not.

I'm starting to see the same behavior on a friend's site in an unrelated niche.

[backdraft7]

I'm glad someone else is seeing this. I know there is an ongoing Wordpress BFA in progress and it's been going on for 2 weeks now. I suspect that the perps have made their way into MANY vulnerable systems already and now it sounds like they are using vulnerable home PC's too to build towards a super attack.

Read this article: [zdnet.com...]

[brokaddr]

backdraft7, I don't run Wordpress on any site I work on. I do however, see WP exploit testing rather frequently - those are easy to spot.

The zero second visits, at first glance, look legit as if it's an actual user hitting the site, then leaving right away.

[modelman]

To put that another way, Brokaddr, if I go to "Audience" -> "Network" with keyword as my secondary dimension, my number one result is:

Service Provider/Keyword/visits/pages per V./Duration/New V./bounce
microsoft corp/(not set)/15,327/1.00 /00:00:00/99.99%/99.83%

and if I set the secondary dimension to "Flash Version" I get
microsoft corp/11.4 r402/15,259/1.00 /00:00:00/100.00%/100.00%

2100 of those 15k visits come from Lincoln Nebraska with that Flash version and they are also from msn.com

I can look up "audience" -> "Technology" -> "Browser & OS" and get the same bogus traffic all neatly packed into "Mozilla Compatible Agent"

This led me to think it might be a vulnerability in that particular version of Flash that only worked with mozilla on the msn network...sort of a multiple weak link scenario, but after you've barked up enough wrong trees they all start looking the same.

4300 of those 15k come from Hialeah Florida, same flash version, agent and network, and recently they started coming in from Deerfield Beach, Tampa and Gainesville Florida. There are many more with the location "not set" (5246 bouncers, all with Flash 11.4 r402) but what really spooked me was when I sent out a newsletter that I was going to New Jersey, they started coming from Paterson NJ, only five miles from where I briefly visited. At first I thought it might be me, but it started before I got there and continued after I left. Just a coincidence, I guess.

So I checked my server logs and saw a low level attack against my root ssh and ftp originating from the Republic of North Korea embassy across the street from the United Nations Building in Manhattan. It used two different IP addresses simultaneously. This seemed totally unrelated, but had the same frequency.

Even if the keyword is not set, it brings down the relevance of that page for any keyword. If this is a malicious attack, and not just some poorly behaved code, then the intention is clearly to reduce my organic search traffic. It has certainly done that. Mostly, I just want to know what's causing this.

[modelman]

A closer look at my server logs showed me that my problem was probably bingbot and BingPreview from a wide variety of IP addresses. I then found the following post on Google Groups which reported identical symptoms to what I was seeing:


"When I investigated an increase in direct traffic on some of the sites I run I saw that it was largely due to a big increase in "Mozilla Compatible Agent" browsers. Digging deeper into the data I see that of the roughly 29,708 "Mozilla Compatible Agent" direct visits to one of the sites from mid December to early February...

29,623 had Microsoft Corp as their network provider
29,623 were Windows 7
29,624 had a screen resolution of 1024x768
28,659 had Flash version 11.4 r402
29,670 were coming from the US or "(not set)"
12,952 came from Hialeah, FL

Hence it would appear a bot is not being properly excluded from Google Analytics reports."

[brokaddr]

I have BingPreview listed as a bot in my reports (not GA) - so my 0 second visits are definitely not related to Bing/Microsoft.

In your case, you could simply filter the "BingPreview" user agent to stop this activity, I would assume?

[Robert Charlton]

Possibly of interest for followers of this thread... we received a post in Google SEO that reminded me of this discussion. It was a report of a browser testing site that's running a script on distributed computers.

It leaves a footprint similar to what's described here, and it was similarly messing up the poster's AdSense impressions, etc. As the OP described...

Footprints, something to look for:
- many visits in a relatively short amount of time with a 100% bounce rate
- visits extremely varied in terms of browser used and mobile vs desktop.
- IPs and referrers are of no consequence, the IPs change between visit groupings and many IPs are used per event
- entire page is loaded including ads
- all hits will be to the same page but each time the script is run any page can be selected

Magnitude of the effect and other signatures not the same. The thread is now in Webmaster General, and I've cross-linked it to this one. Perhaps the discussion will provide some clues....

Identifying sources of unattributed, direct traffic that skew stats
http://www.webmasterworld.com/webmaster/4600519.htm [webmasterworld.com]

[Robert Charlton]

I thought I'd kick this thread up with a reference to a new and possibly related discussion...

TECH UPDATE: Bots as Browsers Using JavaScript
The Bot Detection Game Has Changed
http://www.webmasterworld.com/search_engine_spiders/4619880.htm [webmasterworld.com]

[netmeg]

Interestingly, the same site that was hit when I first responded to this three (18 mos or so ago) is now being hit by another bot with a different footprint. Still trying to narrow this one down, but it looks more like the one in the Webmaster forum thread (boy howdy I wish there was an easy way to put all these in one place)

[Edge]

Netmeg, my bot net is back as well. Simple for me to make it invisible again.

I'm wondering if there is a WebmasterWorld connection..