Logs Show Surge, but Not Human?

[rogerd]

On one site I work with, I've seen traffic go from 10K visits/day to 40K. The additional traffic looks human at first glance - it is captured by Google Analytics, It comes from diverse consumer IPs in the US and Europe (but not Asia), and the bounce rate is high but one out of ten visits or so loads another page.

On the non-human side, we have all of the traffic coming with no referrer, and it is all focused on a few pages that are hardly viral linkbait and would get one or two views on a good day. It's all IE (spread among 6 - 9), and a range of screen resolutions that look unusually aged (e.g., 1024x768).

Anecdotally, I've heard of a few other sites seeing this kind of traffic, but nobody knows what the purpose might be. It's not scraping, as it's the same pages that get hit. It's not intense enough to be an attack to take the site down, nor is the site likely to be the target of miscreants.

The level of traffic has gone up and down, but it's still happening.

Are any of your sites seeing this, and do you have any theories?

Any thoughts on screening this out of Analytics? It totally blows up time period comparisons.

[netmeg]

Well I tried Edge's list, and it didn't match up with the ones hitting mine.

Roger, I contacted Compuware as well, and they escalated it, but came back and said no, it wasn't them. They say that they identify themselves in their User Agent.

Someone from Google popped in the Google Analytics help forum in Google Groups and said they were looking into it. They have more resources than we do. But don't hold your breath for anything happening right away.

Right now I've it not serving AdSense or analytics codes to anyone who comes in on IE with no referrer, and hoping for the best. All I can do.

[macavity]

Is there any consensus at the moment on the best way to deal with this at site level without impacting legitimate traffic? Is IP blocking the best approach?

[macavity]

Aha, netmeg, our messages crossed. Looks like IP blocking (using a shared list) may not be that effective then.

[Mario155]

Netmeg, can you please send me Edge's IP block list? thanks

[Edge]

Ok, my original ip ban script/list worked well at first however the botnet has been slowly building back with new ip addresses.

I can get much of the balance back to my site by running my ban script for a few hours but as soon as I stop progress is reversed.

This thing is huge or very dynamic. I do have some of the early captured ip’s returning however, I think most are not returning.

I have started running seoskunk ignore no referred visits script on effected webpages. Thanks seoskunk.

Mario155 - my ip list will not help you.

[Mario155]

Thanks for the info, this is horrible, it is ruining my site. It has been going on since Feb 21st. It must be some hacker doing it for laughs, I cant believe it has not been mentioned anywhere in public yet.

[netmeg]

Yah, mine started Feb 21 as well. My traffic has slowed, which is good, but it's definitely still going.

(Every time I hit this forum, I hear the music from the first Terminator movie in my head)

[Mario155]

My bot traffic has slowed as well since 2/21, but it is still coming in. My site has been crashing on and off today, but maybe it is not related to this, I don't know.

It is still causing an issue with bounce rate. Also when I got the finalized numbers for the month of February from AdSense, it was almost 20% less than what the estimated figures were for the month. The final earnings are always lower than the estimated, but never by this much. They must have noticed the bot activity.

[Webwork]

What's the motivation for this attack?

Is there buzz about sites, other than those running Adsense, being hit by this attack?

Follow the money? Ex. If "only Adsense" then a) competitive site that's been tanked; b) this is "the new SEO" (nuke competitiors by botnet); c) punish G; d) "This is a test. This is only a test." For what reason? Proof of concept? How to massively expand DDOS, not from a site take-down perspective but from the worst wound with the least commitment of botnet resources? Kill the moneymaker page(s) and cause the most pain? Extortion notes to follow?

I'd think about defenses from the perspective of "this is a test" and there's more to come.

I'd also . . cough . . cough . . look for the new(est) hosting provider(s) that offer services designed specifically around IP blocking. (And, if I was a hosting provider, I'd consider making this "offer" part of the marketing package, i.e., we offer upstream botnet protection from all known attacks . . )

Wish I was more tech savvy. Based on my limited knowledge the best temporary solution that may exist is for widespread IP list sharing. Are there agencies/entities that serve to collect lists of zombie machines and do they share their lists, especially IF you contribute to their lists?

[Webwork]

Hot off the presses.

AT&T Reluctant to Respond to F.C.C.'s Request for ISPs to Play More Active Role in Policing Zombie PCs / Botnets Operating on the ISP's Network. [cio.com.au ]

So, what now? What IF people start forwarding Botnet info to agencies that collect this info FOR THE PURPOSE OF proving that the ISP's are PART of the PROBLEM . . handing off the data to the Class Action Lawyers . . who can represent all the websites that are being affected/hurt/takedown whilsts AT&T et. al. watch?

Follow the money: ISPs make more money allowing their client's PCs to run infected than they would otherwise by sending out notices, disconnecting homes, etc. Better to let Bob and Betty run their infected circa 2001 PC and collect their monthly fee (Hey, we've got the network capacity to allow for thousands of fee paying Bob and Bettys!).

I sort of get the problem and why it persists. Botnets, to ISPs, are just a small cost of doing business. No one wants to step up, be the first to intervene. They might lose that lovely monthly rebillable.

Clearly this issue - BOTNETS - requires a multi-pronged response. It isn't just a tech issue. It's a policy issue, at the highest level. It's a business issue, one for which everyone affected needs to start making noise, contacting their Congressperson, their Chamber of Commerce, etc. This IS a threat to business. This issue is likely one that will NOT be solved at the hosting level BUT ONLY at the ISP level with ISPs disabling accounts, etc.

It's also an international issue and one that will likely require the passage of treaties - SOON, as if it's a PRIORITY - as botnets are clearly a threat to all forms of international commerce, national security, etc.

It's a bit daft that it's 2012 and we're having this discussion, in our little forum, about an issue - BOTNETS - that a) has been an known issue for a decade; b) is clearly beyond our individual means to remedy/addresss by a tech fix alone; and, c) is threatening to so many, in so many ways.

Call the F.C.C. or somebody "up there" whilst you also labor to apply a patch at the local level.

[netmeg]

Mario155, you are risking your AdSense account if you leave ads on the pages that are getting hit. I'd take it off.

[rogerd]

Since these are consumer IPs, by and large, I guess it's no surprise that the IPs don't persist over time.

I'm beginning to think the only viable approach is to file abuse complaints with the ISPs to force them to see what their customers are doing - either voluntarily running some kind of script (e.g., Gomez Peer Zone) or harboring malware they don't know about.

[tangor]

Mario155, you are risking your AdSense account if you leave ads on the pages that are getting hit. I'd take it off.

And that might be the entire reason, in a nutshell.

[Mario155]

AdSense would ban my account even though they are aware of this? I saw on the AdSense forum that people have already told them of these attacks. I am not really noticing anymore huge increase in revenue on the 2 pages that these bots are visiting. There was last month, but AdSense adjusted it for the finalized earnings.

Do you think if AdSense has to do another huge adjustment on my earnings for March it may trigger a site review?

[macavity]

As someone commented previously, I'm surprised this hasn't generated more discussion elsewhere or hit some tech/security news sites. Perhaps I'm just overestimating the scale of the problem but it *feels* widespread.

[ken_b]

AdSense:

If you have AdSense on these pages, I'd suggest taking it off immediately.

I took AdSense of my homepage (the only affected page on my site) as soon as I noticed this traffic spike.

Still, the home page channel had chalked up WAY more earnings than normal.

In the end of month earnings adjustment for February AdSense took back all but a few pennies that channel was credited with for the time between when the attack started and when I took the ads off. (I'm fine with that, advertiser certainly shouldn't be paying for this traffic)

I don't get it, what can the motivation be for this traffic?

In my case, I can't imagine it's a competitor to my "non-ecom" hobby oriented site.
.

[rogerd]

The types of sites being hit are all over the spectrum. A specific, purposeful attack seems unlikely to hit such a diverse group of sites.

[netmeg]

As I have explained elsewhere re Adsense - it doesn't matter whether or not it's your "fault" or your site is being attacked and out of your own control. The simple truth is, it's bad traffic, and it puts the advertisers at risk if it starts triggering clicks. Or if you are getting CPM traffic that pays by 1000/impressions. There is no fair here - the advertisers don't want to pay Google for this, and Google isn't going to pay you if they don't get paid. So take the AdSense off the pages that are getting hit. The bottom line with Google is always going to be keeping the advertisers trusting the system.

[staffjam]

I've sent an overview of what's happening here to some tech editorial contacts i have - hopefully someone will pick up on the story and get a bit more publicity for this. Google have apparently been looking into this for over a week now and haven't come back with anything helpful.

[freelistfool]

I had something similar happen to one of our sites. However, my traffic comes from several different browsers (mostly Firefox) so it might not be the same issue...but I'm posting our experience in hope that it helps.

On 2/21/2012 we noticed a spike in direct traffic with the following characteristics.

1. Hitting only the home page
2. Accounts for more than 90% of the traffic to the page
3. Has a high bounce rate (over 75%)
4. Coming from a large number of different IPs
5. Coming from several geographic locations
6. Used several different browsers

Once we isolated what a bad hit looked like we traced several of their IP addresses in our logs to see how many times a single IP was hitting the site. During this process we found a very small number of hits from those IPs with a referrer. The first hit to our site from that referrer was on 12/9/2011. The referral and direct hits started out very small but the direct traffic grew exponentially (doubling every two weeks).

We blocked the offending referrer and it cut the direct traffic in half within about 24 hours. During the process of reverse lookups and blocking IP addresses on the direct traffic we found that the majority of hits were from the same country (Bangladesh in our case) and almost all hits with a referrer were from the same country. So we just blocked all traffic from Bangladesh. Within 72 hours our traffic was back to normal and the direct hits from all other geographic locations was back to normal too. As of today, the attempts to hit our site from the bad IP addresses has dropped from several a minute to only a handful a day.

Now we're hoping it doesn't pop up from another country or referrer. Hope this helps.

[Mario155]

How does AdSense or the advertiser know that it is not valid traffic though on these pages? Because of the high bounce rate?

[netmeg]

They always know; they have a ton of ways to watch that stuff.

[lucy24]

Is there buzz about sites, other than those running Adsense, being hit by this attack?

Never mind about buzz. The question is at least answerable within this thread. Has anyone here been hit who is not running AdSense? If yes, scratch that as the significant variable. Conversely, is there anyone running AdSense on a big site who has not been hit? If yes, AdSense isn't the only variable.

fwiw: I tend to get hit by most bots that are going around. This one hasn't touched me. (I would notice.)

[Edge]

AdSense isn't the only variable.

Perhaps, what I do know is that it seems the browser is almost always MSIE 9.

[Seb7]

This code will block IE with no referral.
Place within the header tags:

<script>
if (navigator.appName=='Microsoft Internet Explorer' && document.referrer=='') window.onload=function(){document.body.innerHTML = "";}
</script>

[netmeg]

Yes, people are getting hit that don't run AdSense. And yes, some of my bigger sites that run AdSense haven't been touched.

I thought about blocking IE with no referral, but (at least up till this happened) I was trying to establish a brand, and I don't want to block real traffic if I can help it. Right now I'm just not showing them AdSense or Analytics. Plus I blocked all non-en browsers.

[lucy24]

Do these particular robots read javascript? Consistently and reliably? Sorry. Can't remember if that came up in the previous five pages of this thread.

I've got something similar in my htaccess: the middle range of MSIE is routed to the "I don't like your face"* page if they don't come in from a select range of Canadian IPs or a few other reasonable exception.

I'd be leery of "navigator.appName" by itself though. Javascript is pretty minimalist when it comes to IDing browsers. Might throw in a second test using "navigator.userAgent".


* Which now says "I'm sorry, but the server thinks you're a robot" --working on the assumption that nobody but an innocent human would ever actually see the page. Like a 403.

[Seb7]

Hi lucy,

Do these particular robots read javascript?

Most bots dont run JavaScript, but this one does (as analytics's is capturing screen resolutions), and my gut feeling is that it actually is an IE window being run on autopilot. So using some javascript would be ideal for this situation.

I'd be leery of "navigator.appName"...

Since were are only looking for an IE browser, I only need to use the IE javascript version of getting its user agent.


Add this to only be non-english versions:

.. && (navigator.userLanguage.indexOf('en')<1)

Modifiy to this if you want to redirect instead of abort the page.

.. location='www.example.com/robot.htm';

[Edge]

Ok, anybody interested who has a clear and significant history on WebmasterWorld - I have setup Fluid Dynamics AXS script to record details about this BotNet on one webpage of mine. .

Sticky me, I will get you the no-login required link to the details.


This webpage normally does see not much traffic, so I estimate that 95% + of the recorded data is this botnet thing. I have cleared my ip block list, fixed GG Analytics, AdSense and is letting this thing run. I started the script Friday at about 9:00om EST.

[Edge]

Yes, people are getting hit that don't run AdSense.

I wonder how many of these pages are adwords landing pages...