Google Traffic Throttling - revisited

[tedster]

The topic of Google traffic throttling came up a good number of times over the past two years or more. Some doubt that it happens - others are convinced but have trouble breaking through the wall of other's doubt.

Last year I saw the analytics for one site that showed a drop to zero Google traffic every day at around the same time - it stayed at zero until the next day and then it cycled again. I also saw another major enterprise site that only had first page ranking for a specific 4-hour period every day. Both of the cases persisted for many weeks, but both eventually "returned to normal".

Just recently someone else contacted me with the graphs to show something very strange. Three months of nearly level Google traffic (unnaturally level in my experience) followed by a huge spike that lasted 4 hours - I'm talking up 1,000%.

So I think it's time to look at this topic again - seriously. It's not a fantasy. Something Google does every so often looks like traffic throttling or rationing. Here's a post from just today:

backdraft: I get these "windows" of sales only minutes apart, then hours of nothing. It's as if Google can turn us on and off while still showing our site in the SERPS. That is very strange and an experience shared by other sites over in the UK. I am in the US.

[webmasterworld.com...]
Here's a small sampling of past and recent discussion:

1. Google Traffic Throttling - where are we on this? [webmasterworld.com]
2. Time of day rankings changes [webmasterworld.com]
3. The Yo Yo Effect - is it now getting worse? [webmasterworld.com]
4. Huge drop in my blog traffic [webmasterworld.com]
5. Google Toggles our High Ranking On/Off Again and Again [webmasterworld.com]
6. One day spike in Google referrals [webmasterworld.com]
7. Is there a threshold for google traffic? [webmasterworld.com]

And there's more where that came from.

It seems like it's a lot harder to pin down now, and it was never easy. So what do you see? Is anyone wrestling with this on their own sites? Any ideas how it is being accomplished?

[Whitey]

Is this an enterprise site thing only , or across the wide range of sites?

[tedster]

I've seen the signs of it (whatever it is) on small business sites as well as enterprise sites. It's not all that common, but as the past two years of reports show - it's not exactly rare either.

It's not even clear to me if the "yo-yo" is part of the same phenomenon, but it seems to me that the only real way to throttle traffic is to change the rankings.

In our current thread for discussing Google Updates and SERP Changes [webmasterworld.com] there are a good number of people who mention that their traffic seems to fluctuate from non-targeted countries. So even though the total traffic stays pretty stable, slaes are on a roller coaster. I'm not sure if this mechanism plays in or not, but it does sound suggestive.

And then finally, there's the reports of sudden spikes. That sounds like long awaited good news for the business owner who has been "throttled". Maybe it is just that - a test of seeing what happens when the limiting factor is removed.

But that's just my guesswork for now. I do not work with any sites that are seeing this behavior, so I can't investigate it very deeply. That's why I started this thread - to ask webmasters who think that their traffic might be throttled to start a digging party into their analytics. After two years, it would be good to finally get somewhere solid on this topic.

[Lapizuli]

I haven't seen these kinds of delays on my tiny sites or pages, that I can recall.

Could this be an artifact of reporting delays with the stats programs? As in, it's the reporting that's bottlenecking, not the SERPs?

If it does exist, my first instinct is to think it's nothing nefarious, just Google doing some form of systematic testing, or down servers, or a bug.

In the event that it's server-related, it could be useful to know the locations of the websites this happens to, and of their primary markets, and how geographically narrow they are.

[chrism]

Not sure if we're seeing something similar on one site, but ours is a much longer cycle - around two weeks.

The first week will have several hundred visitors from google per day, the second will have nothing, with the exception of two long tail keywords which generate the only google traffic.

Initially we thought we had a penalty, or at least an automatic filter of some sort, but it seems odd that all but two keywords disappear completely.

On the good weeks, our traffic comes from a few 'big' terms (about 40% of overall traffic) and the rest from a few referrals from each of 100+ phrases.

Edit: just an additional detail - the exact cycle is 7 days of normal traffic then six days of very low traffic and then it repeats.

[cls_wired]

Tedster, i'll try to give several versions in assumption that this is Google bug.

1) Hardware or network problems on Google side. The server (or bunch of servers), hosting the part of index file with problem site in it, may to be in troubles with network latency and information can not get to ranking server in time. Due to periodical properties of network (route tables updated by cron, etc), we can see this periodical problems.

2) Or, may be, it is consequences of joint work of some inner Google programs. For instants, when site is in top, and gets a lot of traffic, one of antispam programs can find strange users activity, and pessimize the site, than, the next programm, which looks on other site parametrs, can unban it. And loop is done

Certainly it only assumptions, but knowing a bit how search engines do work from inside, I can tell that if it is a bug, the reason should be very similar.

[karter]

As a matter of interest do all these sites use GA

[Jez123]

Interesting. My own site is probably too small to track these sort of things but it's the sort of thing I am seeing. SERPS steady and good for main terms but traffic down and useful traffic at an all time low.

[chrism]

karter - in my case yes GA is used on this particular site.

[londrum]

i reckon this is the way its supposed to work. google drops sites in and out of the SERPs depending on what time of day/week it is. it makes sense if you think about it.

if you look for train times in the morning, then they probably want today's train times. but if they search for train times in the evening, then they probably want tomorrow's train times. so google will return different results depending on what time of day the search is made.

[drall]

no GA, our biggest site has been throttled for years.

[Jez123]

@ londrum. So do you think that it might be similar but geographically throttled do you think? As when I see my SERPS they are always good but peopole in other parts of the country / world may be seeing totally different results?

[Reno]

in assumption that this is Google bug.

I'm not sure that it is a bug, and I'm not sure that it is bad. Sure, I'd love to see consistency as long as the consistency benefits me. But suppose "consistency" knocks me down a couple pages and that same consistency keeps others steadily above me. Well, then I don't like it. :(

The WWW is well beyond what anyone could have predicted 20 years ago, and it grows every day in every way. Maybe, just maybe, "throttling" is Google's way of distributing traffic to sites that are pretty much equal. I don't know ~ I DO know that I believe it is real because it begs credulity to see such consistency in numbers from day to day simply because of coincidence.

So I go back & forth, and leave it to others to make the argument for its demise. In the meantime, I work on my sites to naturally get better traffic (in quantity & quality), and given how many people at this very forum report falling through the floor because of this that or the other, I see those months where the numbers are about as good as the months before as being a small victory of sorts. Or at the very least, it may be the best I can hope for under current circumstances.

.......................

[tedster]

karter: do all these sites use GA

The two cases where I saw the analytics did not use GA. One was on Omniture and the other on something I had never seen before.

drall: no GA, our biggest site has been throttled for years.

You may be just the guy I'm looking for! Do you have any insight into HOW Google might be throttling your traffic?

[edited by: tedster at 9:47 pm (utc) on Dec 30, 2010]

[drall]

Tedster, it's very sophisticated.

How? First lets look at the data.

- PR 8 site.
- 1.1 million searchs per month.
- 120,000 pages handbuilt over 12 years.
- 36,000ish searchs per day

Up until 2005 this sites traffic would flux with the workweek. So lets say it looked like this.

- Sunday 50,000
- Monday 29,000
- Tuesday 31,000
- Wednesday 30,000
- Thursday 35,000
- Friday 39,000
- Saturday 47,000

Daily deviation would be as extreme as 50% sometimes. Totally natural. Then something happened slowly over time.

Until the week started looking like this and has stayed this way since.

(last weeks actual log)
- Sunday 36,425
- Monday 36,527
- Tuesday 36,487
- Wednesday 36,427
- Thursday 36,430
- Friday 36,428
- Saturday 36,454

If we have a large amount of new IBL's come in, nothing changes.
If we pump out a new section with tons of content, nothing changes.
If we remove large amount of content, nothing changes.
If we do anything, nothing changes.

I was convinced we had been hacked at one point but our security team shows an all clear.

Now the how part.

As with any good webmaster I have understood the power of a raw log file from the beginning I know what I am looking at and it is simply jaw dropping. The level of computational power Google must use to throttle boggles my mind.

Entire sets of randomly rotating serps that are tailored throughout the day to throttle us. Lately as I have been pointing out in other threads pages with social networking strength have been rocking for us.

So much so that some of these pages can bring in 3-4000 visitors a day. What is incredible to watch is after our visit count spikes for some page like that, other pages that normally perform #*$!x visitors per day will all suffer just enough the rest of the day to level us out to that 36,000 number.

My gut tells me that our root domain has been flagged somehow. A daily limit was assigned after team review. That root domain is "in the system" and balanced via tracking throughout the day.

My hunch is that it is a reverse engineered adwords/doubleclick for publishers daily budget planner but budgeting traffic instead of clicks or impressions.

Basically our domain is the campaign!

[Reno]

drall ~ are the numbers from '05 entirely made up, or are they close to right? When I add those up and divide by 7, it averages about 37,250 per day; your current average (based on last weeks log) is about 36,454. So I'm just wondering to myself if Google looked at your numbers over a specified time period, averaged them out, and set that as the marker. Otherwise, my question is, why the 36,000 range? Why not 42, or 39.5? There must be some rationale to the number they set for you, and for the rest of us. My own numbers are considerably lower, but even so they have about the same amount of variation, which is to say, not much. That doesn't make sense and, to my eyes, can only be explained by the throttling effect.

....................

[tedster]

What is incredible to watch is after our visit count spikes for some page like that, other pages that normally perform [thousands of] visitors per day will all suffer just enough the rest of the day to level us out to that 36,000 number.

Thank you for those details - that is the kind of effect I'd expect if something quite sophisticated is going on.

[edited by: tedster at 6:36 pm (utc) on Sep 20, 2010]

[Lapizuli]

Drall, does it keep to that pattern year after year? Season to season? No growth in overall traffic since 2005?

[backdraft7]

Over the last few days our page views went from about 100k per day to 10k...today is seems to be building back up. However, I compared page views this year to the same period last year and they were typically 10k with great sales. This year is can be 100k per day and poor sales. Argh!

I think we had a better user experience when this was all the Google was....[infolab.stanford.edu ]

[londrum]

maybe google needs to do this kind of thing to weight our pages.

imagine that google has got five sites about a similar subject, and they want to factor in user behviour, or something like that, to guage which is best.

that would be a lot more difficult to do if one site had 100,000 visits a day, whilst at the other end of the scale one only had 5,000 visits a day.
just because they get less traffic doesn't necessarily mean its a worse site.

so if google gives them all 36,000 visits a day and checks the stats again, all the sites are on an even keel, and they can better decide which is the best.

[drall]

Reno, the first set of numbers from 05 are just rough estimates from memory. I can digup my logs from then and get a actual number if you guys want.

Tedster, yep! Whatever it is it's extremely complex. I mean to balance that across tens of thousands of different queries per day just for us is way to hard to get my mind around. It has to be some form of realtime domain level balancing. I have tried everything possible to escape it. After 4 years nothing has worked.

Lapizuli, the pattern remains the same. We stopped growing in 2005 in Google organically at the same time that the throttling started.

We have had a few knocks down like mayday but when this happens a new baseline is set. So lets say in 2005 we did 46k per day, now it's 36k per day, every day. Most updates dont effect us but mayday did.

Every one of my competitors which is in the hundreds all got whacked by Mayday. All lost 25-75% of their Google search traffic. As far as I know, NONE increased.

Whatever it is, it's real. Our best guess here at the office is Google wanted to spread around the traffic to keep as many adsense publishers happy as possible and to also not have any one company solely dominate any given niche. Also it gives the little fella a chance.

It's all good, there are limits to everything:) I made a post about this a couple years ago but not many folks believed me. Seems like more are seeing the same thing now which is comforting in a sick sort of way.

[edited by: drall at 7:35 pm (utc) on Sep 20, 2010]

[thelostagency]

I have seen it a fair bit but then sometimes i've drill into the data and found that it was from images.google.com or google.co.nz or one of the other google domains

seen very weird stuff showing up in Google Webmaster Tools and have noticed some sites almost seemed to have the traffic shaped to fit more of what might be part of a Bell Curve, so more advanced than throttling

[blend27]

drall,

I see the exact same thing on a smaller ecom site, but the Niche Leader.

What interesting is that "SUM of all page views per user sassion"/"Unique visitors" per day balances out about the same over all per day. As if G is sennding less interested people at times during the part of the day.

NO GA on this site either.

[chrism]

Just a thought - could these volumes be set (limited) by what Google believes (note believes :) ) your server(s) can handle before they begin to slow down and affect user experience?

[Reno]

drall, tedster, et al... Interesting thread that only confirms what I've believed for some time. But again my central question is no longer "Is throttling real?" ~ that's been settled to my satisfaction. Rather, it's "How does Google establish the target range?" And if we ever know an answer to that fundamental question, "Is there anything we can do to grow out of that range?" I think when we better understand that, we will see throttling as more of a limitation for others, and less so for ourselves. But therein lies the mystery....

..........................

[tedster]

That was one of my early thoughts, too, especially when the occasional spike is observed and then it goes away.

But it just doesn't make sense in the big picture. There are some very strong servers represented in the above reports. One I know of uses Akamai as the CDN. And I doubt that even Google can monitor traffic so well that they notice a server response slow-down and change the traffic they send in nearly real time.

[Robert Charlton]

There are some market areas where query volume relates to time of day and is likely to correlate, eg, with work hours. One way to check this would be to see if similar patterns can be observed across time zones.

[backdraft7]

With DARPA's hand in Google from the get go (funded equipment), I'm not surprised of the endless complexity of the algo's. I believe it's Google's end goal to make it so complex that nobody on the outside ever figures it out, thus preventing the eventual hacks and workarounds that were once effective on Google. The other goal may to throttle businesses so as to prevent competition....hmmm, ain't that anti-trust?
If you can't tell, I'm having another record slow day...

BTW: I am also seeing this throttle pattern where our traffic and sales numbers fluctuated in a natural pattern, but now (since about May 2010) I am seeing weeks with nearly identical traffic and sales numbers and those numbers are considerably lower than they used to be.

Google's tinkering sure has a negative effect on the economy of those who, like me, are foolish enough to live off the web.

[DonMateo]

Another possibility (except for Drall's case which is pretty convincing of traffic throttling) is that competitors and other sites switch their adwords campaigns on and off at certain times of day, or they hit their daily budgets.

When they're running good campaigns they suck traffic away from you, but when their campaigns are paused you get more visitors.

[steerpikegg]

It's funny that this should come up as I had recently been thinking the same thing. Around August 3rd, our traffic dropped by around 80% and since then, our graph in WMT is literally a flat line. Before this, a natural bumpy curve like you would expect, but now just dead flat. It's like somebody has said right, you are not having any more traffic than this under any circumstance.

This seemed odd to me, any when I looked at our server stats and logs, you can see that daily variation is caused by Bing / Yahoo / Direct and that Google traffic is almost identical every day, so that for me has ruled out WMT reporting being the cause.