Websites Affected By Zombie Traffic - Observations & Analysis

[reseller]

I wish this thread to focus on latest observations and analysis of WebmasterWorld members who do believe that their website(s) are negatively affected by the so called Zombie Traffic.

A friendly remark: If you want to talk about PPC issues please head over to Google Adwords forum [webmasterworld.com...]
If you want to talk about Google business decisions please head over to the Google Business & Finance forum [webmasterworld.com...]
If you want to vent some frustration then head over to Foo [webmasterworld.com...]

[tangor]

This is where I get really confused. Statements like:

"Cutting off my traffic" and "Intimidating my customers."

How, EXACTLY is that done? The assumption is the mere presence of just another hit on the website can affect other hits (short of taking down a server) is an astonishing claim. Either that or a very under-powered host.

OR, the claim is being made for managed and throttled traffic.

[aristotle]

They could be indian or brazilian bots

If they're comingn through proxies, how do you know where they originated?

Like I said before, you should just forget about them.

[mosxu]

@NoobOperator

The good news is that for now humans are smarter than AI...

I wonder how is it possible that Kaspersky is not able to indentify "glakes special browser" ?

[glakes]

@nooboperator

You place a lot of emphasis on zombies simply being bots. I don't see this and GA data closely resembles server side stats and raw access logs within 7 users. That's pretty darn close out of thousands of visitors, and I'm not aware of GA or GSC counting spoofed bot hits from external sources as originating from Google (correct me if I am wrong). For all you have said, Google would have to be the botmaster controlling the botnet to make sure on and off zombie days maintain a similar quantity of traffic and the quantity of traffic gets reported in GA and GSC. If nothing else, Google would have to allow "x" number of external bots before throttling kicks in and real humans evaporate yet still count such fake Google hits as real. This just does not jive in my situation, but in saying this I am not implying you don't have a problem with bots - just that I don't.

Many of us saw 3-4% conversion rates from Google disappear around the same time in the fall of 2015. The fact that this occurred around the same time makes it extremely unlikely that some third party botmaster just happened to target all of us at once. And when Google does major updates, I see conversion rates return to normal, once again suggesting that a third party is not in control of the zombie traffic. Others have reported the same, and it does feel like a filter is turned off when major updates occur. From what I see, Google does a pretty good job at preventing automated queries from being passed on from their domain. Anyway, I'm in an industry so small that I doubt many here even know the products I manufacture even exist.

I would agree that bot traffic can contribute to what some see as zombies. Though, when one starts looking at the timeline of events, different occurrences when zombie traffic returns to normal and evaluating Google's traffic quality compared to other search engines, it becomes crystal clear that Google is responsible and not some third party botmaster. Another thing that does contribute to what some may see as zombies is how Google re-writes titles and descriptions. I've always viewed this as a form of cloaking by Google - showing their users something in a way which we as content creators never intended. I've seen some examples of Google butchering the rewritten titles and descriptions so badly that high bounce rates should be expected. However, I don't see this as a widespread problem in my case.

In the end the metric that matters most to me is conversions. Everything, including what I say, is pure speculation as their is no smoking gun to place the blame on. If Bing and Yahoo can produce more daily sales at 100 visitors then Google's thousands of visitors, I'll come to my own conclusion. And that conclusion is, for the time being, Google traffic is #*$! (whether referenced as bot, human or zombie). I still get purchase orders via email and fax, while being able to reach new customers on Amazon, Bing, Yahoo and on rare days Google. I can't change Google, but I can and have diversified my sources of traffic that allows my business to grow without wasting much effort on Google's shenanigans. Would I like to see Google's zombies disappear? Certainly. But what is working for Google (good profits) may contradict with what works for me (getting rid of zombies).

[NickMNS]

@aristotle

Noob's assertion that they are from Brazil and India come from my comments here: [webmasterworld.com...]

I determined this from the fact that the referrers were from google.co.in and google.br in a few instances that I found in my logs, but this is not exclusive.
A. There is no guarantee that the users are in fact from these location, all we know is that they came those url, so there is a high probability that the user are from those country.
B. These are only two of many examples and users could be coming from anywhere. (note that the google translate example below shows .de).
C. The point of all this was to say that the IP's arise when users use Google services such as GoogleWeblight or Google Translate or potentially other similar Google services. I come to this conclusion since it is explicitly noted in my logs with

(KHTML, like Gecko; googleweblight)

or

66.249.93.65 - - [08/Oct/2016:13:38:07 -0400] "GET /XXX/X/X/XXXX.html HTTP/1.1" 200 36178 "https://translate.googleusercontent.com/translate_c?depth=1&hl=de&prev=search&rurl=translate.google.de&sl=en&u=http://www.XXXX...

Are these bots?

Technically I guess these are bots. Since Google is scrapping the content and then changing it in some manner before showing it back to the user.
Are the initial requests coming from bots, it is possible, but I don't see the point of a bot using Google translate or Googleweblight.

Also note that if you are looking at incomplete or summarized logs you may be missing some fundamental information that would make the requests look very strange indeed. I noticed that that for any single page requested by a user using Googleweblight that Google can use several different IP's to collect the assets from the same page. Eg: html is collected by one IP, images by another. So if some of the information is missing, it could appear like you have IPs collecting nonsensical assets, like only collecting images. Where in fact it is completely logical. I assume that Google does this to speed up the delivery of the page streamlined page to the user.

[aristotle]

glakes -- It's quite possible that a lot of the newer types of bots are using fake google referals, and are also getting better at mimicing the behavior of real humans. In addition, if they come from a botnet, they're even harder to distinguish from humans. I don't use Analytics, but Statcounter is still able to filter most bots out, although more do seem to be getting through lately.

This may be why some people are saying that zombies are bots-- they're seeing bots that are good at mimicing humans. So the discussion here has gotten confused, with some talking about bots and others talking about humans, but both using the word "zombies".

I still think that zombies are humans, but humans that google has identified as unlikely to buy anything in the near term. In other words, google can pick out most of the people who are ready to buy and send them to favored sites. What's left after the potential buyers are removed is sent to less favored sites.

As for identifying bots in the logs, here are a few clues:

-- bots usually don't download images.
-- bots usually don't download supporting files.
-- bots usually don't ask for favicons.
-- bots usually don't execute scripts.
-- bots usually don't accept compressed files.

But as I said, bots are getting better at acting like humans, so those clues may not work as well as they once did.

NickMNS -- I saw your earlier post, but thought you were talking about different IPs than those two google proxies that NoobOperator posted. Those look like entries for bots with fake google referals. GoogleWEBLight probably has its own IPs

[NoobOperator]

@aristotle

If they're comingn through proxies, how do you know where they originated?

Like I said before, you should just forget about them.

How do you know they come through proxies ? If you have info that I don't then feel free to share it.

I usually need good reasons to forget about something. What would the reason be ?

bots usually don't download images

Googlebot, bingbot, yahoo bot, baidu bot, xxxx bot all download images from me. You can test this by going to google images and find all the images they have taken from you.

[edited by: NoobOperator at 6:22 pm (utc) on Feb 12, 2017]

[NoobOperator]

@mosxu

I wonder how is it possible that Kaspersky is not able to indentify "glakes special browser" ?

I don't know. May be you can ask Kaspersky ? At a guess I would say Kaspersky never heard of "glakes special browser" and therefore unable to identify it.

[NoobOperator]

@glakes

Yes I do see zombies as bots. Reason is each copy of chrome is a separate bot and can be commanded to send a web request to any address of google's choosing. Faking the web request origin is difficult if not impossible if done in another way. I have't read google's terms and conditions, but you would have found the smoking gun of chrome being a bot if the terms have something to the effect that it is allowed to send traffic to anywhere even if the user did not instigate it. Such traffic are bot/zombie traffic.

I don't understand why you need to get rid of bot/zombies. They may not do you any good, they certainly will do you no harm.

Diversifying the traffic sources certainly will make google zombies less of a problem. But that is not without cost. Maybe the cost is just as much if you simply paid google.

[aristotle]

NoobOperator:

Hostname:
google-proxy-66-102-8-28.google.com

Hostname:
google-proxy-64-233-172-130.google.com

[aristotle]

NoobOperator
Rogue bots usually don't download images

[NoobOperator]

@aristotle

What does a proxy mean to you, that the address is not owned and operated by google ?

What exactly are rogue bots ? What are the pro's and con's of them compared to say googlebot ?

[glakes]

@aristotle

Let's assume bots are getting through by faking the referrer. Google Analytics and Google Search Console are also counting these hits as coming from Google. Could it be that Google itself is incapable of differentiating bot traffic that spoofs Google's own domain and is erroneously reporting that traffic in GA and GSC as legitimate? I've seen no evidence of this in my case. The vast majority of the bot traffic I receive still originates from hosting providers and makes up a very low percentage of total hits.

I still think that zombies are humans, but humans that google has identified as unlikely to buy anything in the near term. In other words, google can pick out most of the people who are ready to buy and send them to favored sites. What's left after the potential buyers are removed is sent to less favored sites.

I have a similar belief. In my case Amazon is Google's preferred destination to send all buyers to - even ahead of Adwords. I believe it's this form of buyer distribution by Google that has cost me a significant amount of money (negative ROI), severely diminished the usefulness of Adwords and ultimately forced my exit from that advertising platform except for rare testing. Since I also sell on Amazon I can see the sale shifts during major updates and noted one recently in the Google search updates thread. These are always short lived though and normally it's only a day that I see equally weighted/disbursed (reasonably "fair") sale patterns from Google. It's possible that during these major updates Google may lack the computing power to simultaneously push out the update and interpret/redistribute buyer traffic. I know there are others who have attempted to discount/question the theory of Google identifying buyer traffic in the past, but it has been part of Adwords for years. In other words, at least Google believes they have the capability to identify buyer traffic and sell it for a premium. The same technology that identifies buyers in Adwords would almost certainly be used by Google in organics.

Assuming Google does select its preferred destination to send buyer traffic to, why is Amazon the preferred destination in my case? Amazon is a large and trusted brand, which is next to impossible for any smaller businesses to compete with on those two points. Having more detailed product information, product photographs, video demonstrations, downloadable PDF instructions, better product pricing and lower shipping expenses on my website are not enough to overcome Google's preference in where they send buyers to. Being mobile friendly and having full SSL makes no difference either. Amazon's domain crowding exacerbates the problem but such crowding may be a tool used by Google to further ensure buyers are sent where Google wants them to go by limiting a users choice to visit alternative websites (including the manufacturers of products themselves).

The question of why Google does this is perplexing. Is it a glitch in Google's AI, for a better user experience or is it for Google's own gain? Arguments can be made for most theories out there and no smoking gun exists to either prove or disprove the many theories behind zombies and I doubt any hard facts will be found in the future to put this issue to rest. But I am of the opinion that Google controls their search engine and the zombie traffic patterns some of us see. I also believe there is little that I can do to change the reasons why Google wants to send buyers to Amazon or other destinations -vs- directly to my website where buyers can get more information, better product pricing and more affordable shipping. Should I start buying links, keyword stuff pages, etc.? That's not my style. About the only satisfaction I get is when someone buys something from me on Amazon and contacts me afterwards saying they could have gotten the same items cheaper on my website. My response? Why didn't you buy it from my website then? This does happen and underscores the reason why price does matter to those searching for products. In fact, for product searches, ranking better pricing for the same product should be part of the user experience if that's what Google is after. This is part of the reason why I don't think user experience has anything to do with the reason Google sends my site zombies most of the time.

[aristotle]

What does a proxy mean to you, that the address is not owned and operated by google ?

I don't understand your question. The hostnames I posted say that those ARE proxies and that they ARE owned and operated by google

What exactly are rogue bots ?

Rogue bots are bots used for nefarious purposes. Their creators might want to copy your site, or hack your site, or sabotage your site etc

[NoobOperator]

@glakes

Let's assume bots are getting through by faking the referrer.

If each copy of chrome is a bot, they have no need to fake the referer. Chrome will produce genuine fake-traffic with modified user agent string from the user's IP address. Therefore google analytics is justified to count it as a unique source of traffic from a real genuine address.

[aristotle]

glakes -- I understand your points. Like I said, I don't use Analytics, but I suspect that some of the newest most-advanced bots might be able to trick it.

I don't think that Google Search Console uses information from our servers, so it should be reliable.

[NoobOperator]

@aristotle

I don't understand your question. The hostnames I posted say that those ARE proxies and that they ARE owned and operated by google

I was just wondering if those addresses were operated by indians and brazilians. I feel much relieved they are not. But I still don't see the reasoning to forget about them since those addresses interfere with my sales directly. For instance if I block them, google will pop up a chrome warning saying my product could be dangerous for each customer who wants to buy from me. Secondly, appearance of those addresses in my log always precedes transitioning of human+zombie traffic to predominantly zombie traffic that is really bad for my sales.

Rogue bots are bots used for nefarious purposes. Their creators might want to copy your site, or hack your site, or sabotage your site etc

These bots are the zombies ? I haven't seem many of this kind of zombies for my site. I would find this kind of zombies rather exciting because they would do something, anything. The zombies I am used to just play dead all the time.

[mosxu]

@NoobOperator

What pissed you off so badly to bring such information forward? Not even Kaspersky can indentify it, or maybe Kaspersky is holding other company stock and therefore?

[edited by: mosxu at 9:12 pm (utc) on Feb 12, 2017]

[NoobOperator]

@mosxu

Nothing is pissing me off. I am here talking about zombies and when they are most visible.

I am not sure what Kaspersky has to do with any of this. Are you trying to say Kaspersky should be able to identify the bot in Chrome ? I don't see why they should. Google has more money for lawyers and Kaspersky will get crushed if they are that stupid.

What is it with you and Kaspersky anyway ? You are their rep same way as some other here who are google's rep ? Personally I wouldn't recommend Kaspersky. From first hand experience, I would say all the big US antivirus scanners are way better and more reasonable. I would recommend MS Windows Defender and Symantec. I am not their rep but their scanners are better able to detect real threats and less likely to manufacture fake ones.

[mosxu]

@NoobOperator

I am not a REP only thing is that it should be easy for any antivirus to identify "glakes special browser" unless they do not want to do that.

I am not disputing your theory by the way, really appreciate it.

[glakes]

@aristotle

My GA, GSC, server stats and raw log files are statistically close in reporting traffic from Google. Of the minor differences, some are surely bots but the percentage is so low and this percentage has not really changed since the pre-zombie days in September of 2015. I've mentioned it before, but I'm in a very small industry. Most here probably don't even know my industry exists, and if I were running a botnet I certainly would not waste resources on anyone in my industry. The number of competitors I have is so small I can count them on one hand and the market for what I sell is not widely consumed by the average business or consumer.

@nooboperator

Most of my sales are the result of B2B transactions and many of these buyers are still using IE or whatever their network admins installed on their workstations. Chrome has a marketshare of 43.07% as reported by stat counter. Why aren't the other 57% of Google users buying or are you also suggesting that the web browser is also being spoofed in addition to the referrer? I'm not saying your theory is false, but I don't see how it applies in my case. Even if I factored in 100% of Chrome users providing spoofed referrers, my sales from Google traffic are far worse then 50% less than pre-zombie days without any new competition or change in ranks. And keep in mind that during major updates, I see normal traffic and conversion patterns from Google the day of the update and then traffic returns to zombies quickly afterwards. To me this suggests that my zombies are either AI or algorithm related. Thoughts?

[NickMNS]

@glakes have you seen a change in sales, regardless of medium, since September 2015?

@Aristotle

GoogleWEBLight probably has its own IPs

It is very easy to test whether these are or are not the Googleweblight IPs. Simply go the googleweblight website enter your url. Then check your logs, it may take a few attempts as there are several ips that are used but you should see those IP's. I haven't tried but I will when I have a free moment.

[NoobOperator]

@mosxu

I don't think scanners care about user agent strings. But then I don't use kaspersky and no idea what it does. The online version of their scanner likes to list my product as not-a-virus-downloader about 50% of the product releases. So I have low opinion of them since nothing in my product downloads anything. They just want to troll me for a reaction.

Glad you are not a rep. Reps are dull, unfocused, unknowledgeable, wishywashy and always producing no value. They are like zombies but are alive.

[NoobOperator]

@glakes

Yes, both the referrer and the browser id can be manipulated on the web request to your site. Like I said before, you can change the browser user agent string to anything you like and suddenly you appear as a different kind of user in logs.

Given their wooden-ness, I don't think zombies need AI. They are just sent to you as fillers for time. My own assessment of zombies is whether they interact. Human traffic are quite different. They do all kinds of things. When there are long stretches of time with no interaction, your site would have been infested with zombies. I further assess zombies by the business cycle I am in. If I made some sales and google came have a download, I know it's zombie time ahead. Because the same thing happened before and happened before that, and before that.

[glakes]

@NickMNS

Conversions from Google traffic fell sharply the last week of September of 2015. Then for about a year I would see three or four zombie days a week. These days its more like six or seven zombie days a week. On a good non-zombie day, traffic would convert comparably to Bing / Yahoo and Google's pre-zombie days. Total Google traffic has steadily risen since 9/2015 while their traffic quality has gotten worse. Ranks for buyer keywords have remained steady during this time with no new competition, though I now do compete against myself with my Amazon product listings. On some buyer keyword queries my Amazon product pages do appear in Google above my website and some below, but I made sure to use watered down descriptions on Amazon to keep them from overtaking many of my buyer keywords. This may change as I evaluate the best path forward. I guess I always held out hope Google would remedy this zombie situation and Adwords would become viable once again - allowing me to produce sales at a cost below the 15% transaction fee on Amazon. But as time has passed, so has my hope that smaller players can compete in Google against a super affiliate.

@nooboperator

For all the zombies I see from Google, 95%+ of the traffic they send would have to be spoofed. I just don't see this applying in my situation. I do check my ranks from multiple IPs and little moves in my small industry. It's been this way for years. My guess is that Google identifies the buyer traffic and my listings in Google are never seen. Surely people would want to buy the same products cheaper and have it shipped to them faster also at a lower cost.

[NoobOperator]

@glakes

In terms of pricing, I believe in making it expensive to increase respect, credibility and sales and it appears to work for me. But in my own buying, I will pick cheap.

If the traffic you get are not doing the buying then they are zombies, especially if they don't interact in any way. You can change your site to force some interactions. That will give you a better idea of what you are dealing with. Put up a button that says: free naked lady for every order and see if you can get a rise out of them.

[NickMNS]

@glakes
Thanks for the response, it is clear but you didn't answer my question. Overall, not specific to Google, Amazon, Bing etc... have your sales changed since September 2015?

On some buyer keyword queries my Amazon product pages do appear in Google above my website and some below,

As far as you can tell does Amazon also appear in the Bing and Yahoo serps?

[glakes]

@ NickMNS

Thanks for the response, it is clear but you didn't answer my question. Overall, not specific to Google, Amazon, Bing etc... have your sales changed since September 2015?

Gross product sales increased 147.65% in 2016 from 2015 sales. Much of this growth is the result of supplying product to OEMs and also to wholesalers. Margins suffer slightly in this case, but selling directly to them has minimal if any impact on the retail side (direct to consumer sales).

As far as you can tell does Amazon also appear in the Bing and Yahoo serps?

I'm above one Amazon listing in Bing and Yahoo and beneath two Amazon listings in Google for my primary buyer keyword. This has not changed much over the years except that the Amazon listings in the serps were once category pages and are now my product page(s). In Google these positions don't change on zombie or non-zombie days when checked from multiple IPs across the US. I don't see it change during updates either, when an apparent release from the zombie pattern will occur for a day or maybe two days if I'm real lucky.

@nooboperator

In terms of pricing, I believe in making it expensive to increase respect, credibility and sales and it appears to work for me. But in my own buying, I will pick cheap.

What I produce is definitely not the cheapest on the market but is of the highest quality (better materials and better specifications to meet a broad range of requirements). On Amazon cheap definitely gives one the upper hand. But even a lower priced item that does not work is no bargain compared to one that does. I've always focused on quality, but have patents in the works that will make my now lower priced competitors look like gougers. Should I see my marketshare or interest in my products slip, I'll release these new lower priced items to the public to regain marketshare and make it extremely difficult for what few competitors I have to price match. Innovation never sleeps and any good business plan has mechanisms in place to adjust to dynamic market conditions.

[aristotle]

NickMNS:
It is very easy to test whether these are or are not the Googleweblight IPs. Simply go the googleweblight website enter your url. Then check your logs, it may take a few attempts as there are several ips that are used but you should see those IP's. I haven't tried but I will when I have a free moment.

I'm not sure what IPs you're referring to, or what point you want to make. It wouldn't surprise me if google has 1000's of IPs that it uses, or can use, for GoogleWebLight

[jstacat]

I have experienced 'zombie' traffic that also searches for and clicks out ads. These are robots which are often blocked which then use proxies, and even local isp accts like frontier or comcast.
Comcast will shut down those accts if reported, other ISp ignore complaints. I use HTaccess to block them, and also block ip or range in google & bing ads. Some ad options do not allow ip filtering. Since i started blocking them in adwords & etc, false ad clicks have gone down to -rare-.
Chinese & Russian bots are laughably persistent, using one proxy after another, i can visualise the operator sweating and cursing [they assign someone to hard cases, their humanity shows in their struggles] persisting for months even when they get the 410 gone or 500 msg, whichever they get it is Zed bytes served. these are the zombies that google search sites and will click your ads to get your page[s], even if blocked. and therefore pollute GA.
Google needs to be able to root out this kind of traffic, and it seems that they think it too expensive. Very educational, and can save big bucks.
Looks like i will get into some coding and translate my ability to spot & block into a program. It needs to run above the access to the page interface, like on the firewall level [apache]. Using scripts or php & etc in a web page to defeat them is asking for a big headache, and usually fails.