Google Traffic Throttling - revisited

[tedster]

The topic of Google traffic throttling came up a good number of times over the past two years or more. Some doubt that it happens - others are convinced but have trouble breaking through the wall of other's doubt.

Last year I saw the analytics for one site that showed a drop to zero Google traffic every day at around the same time - it stayed at zero until the next day and then it cycled again. I also saw another major enterprise site that only had first page ranking for a specific 4-hour period every day. Both of the cases persisted for many weeks, but both eventually "returned to normal".

Just recently someone else contacted me with the graphs to show something very strange. Three months of nearly level Google traffic (unnaturally level in my experience) followed by a huge spike that lasted 4 hours - I'm talking up 1,000%.

So I think it's time to look at this topic again - seriously. It's not a fantasy. Something Google does every so often looks like traffic throttling or rationing. Here's a post from just today:

backdraft: I get these "windows" of sales only minutes apart, then hours of nothing. It's as if Google can turn us on and off while still showing our site in the SERPS. That is very strange and an experience shared by other sites over in the UK. I am in the US.

[webmasterworld.com...]
Here's a small sampling of past and recent discussion:

1. Google Traffic Throttling - where are we on this? [webmasterworld.com]
2. Time of day rankings changes [webmasterworld.com]
3. The Yo Yo Effect - is it now getting worse? [webmasterworld.com]
4. Huge drop in my blog traffic [webmasterworld.com]
5. Google Toggles our High Ranking On/Off Again and Again [webmasterworld.com]
6. One day spike in Google referrals [webmasterworld.com]
7. Is there a threshold for google traffic? [webmasterworld.com]

And there's more where that came from.

It seems like it's a lot harder to pin down now, and it was never easy. So what do you see? Is anyone wrestling with this on their own sites? Any ideas how it is being accomplished?

[tedster]

steerpikegg, to the degree that you can, please see what more you can learn about what is happening, and how it is happening. I'd love it if someone could come up with a "catch it in the act" picture of how this works.

[Bennie]

This has been happening for at least 2-3 years and only occurs in some SERPS, not all and it's hard to pin down. I'm sure it has something to do with running two sets of SERPS simultaneously and I wonder if throttling has something to do with this.

[backdraft7]

They should call it the "Hat" algo update; it's like they can now put a lid on whatever traffic they deem fit for your site. Your hat size is calculated by the big black box haberdasher over at Google. It's a scary concept.

[edited by: backdraft7 at 11:03 pm (utc) on Sep 20, 2010]

[petehall]

backdraft: I get these "windows" of sales only minutes apart, then hours of nothing. It's as if Google can turn us on and off while still showing our site in the SERPS. That is very strange and an experience shared by other sites over in the UK. I am in the US.

Have you checked times?

Most sites I watch receive increased orders during lunch hours etc, when people have some free time to surf for personal use.

[backdraft7]

pete - yes, I know exactly what you mean, but no, the spikes can happen at any time of the day or even night...I have a very good bead on my traffic since I use audio doorbells on key pages. I can hear my traffic while I work on other stuff. When it gets quiet, I know something is haywire.
I've been doing this audio monitoring for several years, and it indicated everything was running like clockwork 24/7, right up until May 18th...then a decent recovery, followed by the September 1st upset. I hope the yo yo comes back up!

[Lorel]

I'm not experiencing this throttling problem, in fact my traffic is higher than last year by 7,000 per month (it's mostly info based, not an ecommerce site), nor am I seeing definite spikes on the larger sites I manage that do sell products other than some have reduced traffic this year which I suspected (until now) is due to the economy.

It has to be some form of realtime domain level balancing.

Considering the number of socialists running this country and Goggle cosying up to their policies and controlling most of the Internet couldn't this be an attempt at "spreading the wealth".

I'm wondering if we sort out info vs service vs ecommerce sites if we'll see a pattern, i.e., is it only ecommerce sites effected?

[trinorthlighting]

Has anyone compared this to server speed? I wonder if google is sending traffic to test server speed?

[Whitey]

I wonder if google is sending traffic to test server speed

Good point. Why would Google send more traffic to sites it figures can't handle the traffic well. It makes assumptions about crawl rates , why not align this to traffic rates.

Drall / Tedster - would this apply to the exmaples you've witnessed ?

Another thing , in all of these types of things I think it's good to get behind the algorithmns intention created by Google's search team management . What are they seeking to achieve by rationing results? Is it quality focused ?

[drall]

At least in our case we can rule server speed out.

We have 4 top end servers each handling a different piece, one for images, one for html etc. Best backbone in the world, best DNS, topend security, site is faster then 96% of sites on the web.

I can tell you one thing for sure Tedster, getting more PR doesnt help. Our site went from a 6 to a 8 over the last few years but no change. Deep links, root links by the droves from the biggest sites on the web. No change. Heck sites are using our site in their ads now as "featured on site.com".

I have tried getting links, getting trust, unique content increased, boilerplate decreased, social, higher update frequency, more content, html optimization, seo up, seo down, changing ip's, changing servers, changing whois, decreasing bounce rate to insanely low levels and finding the answer in animal entrails. So far im still in the dark.

[trinorthlighting]

Drall,

I would not rule it out. Google has a speed standard, think of it this way, what if Google is sending you 6 servers worth of traffic?

I think the easiest test to see if that is the case is to increase your server speed by 25% and see if you see a drop in throttling by a % as well.

You know how Google is pushing speed now days.

[Reno]

Entire sets of randomly rotating serps that are tailored throughout the day to throttle us...

...Every one of my competitors which is in the hundreds all got whacked by Mayday. All lost 25-75% of their Google search traffic. As far as I know, NONE increased.

So if you're down AND they're down, where is the traffic going? Has there been a totally unexpected drop of interest in your specific field/subject? Could it be that you're losing a ton of visitors because Google Image Search isn't sending them any more? The drop in your numbers and also the competitors numbers is puzzling ~ where did all those people go?

....................

[Whitey]

Drall - what speed range ( upper and lower ) are you serving pages at ?

The bad part of me want's to say that Google is interfering in traffic delivery for commercial purposes. The good part of me say's it's about SERP quality .

If Google was going to "share" results around and limit traffic to one site , why not others.

Drall , have you seen the same applied to your competitors ? Are they being throttled ?

[backdraft7]

Has anyone compared this to server speed? I wonder if google is sending traffic to test server speed?

Our server is a quad core dedicated box with extra memory and SCSI raid drives. It is easy enough for Google to see our hosting configuration. Speed has never been an issue.

have you seen the same applied to your competitors ?

Yes, I know for a fact that my competitors are suffering the same problems too. (don't ask me how, lol)

I also use MUNIN on my server and if you look at an Apache Accesses by Year graph, I can see wild fluctuations starting in May...previous years had lower statistical kurtosis (peakedness).

[Whitey]

server is a quad core dedicated box with extra memory and SCSI raid drives

But what about the speed of the pages ( the server may be fantastic , it doesn;t always mean the pages load fast ).

Sorry - i'm just trying to discount the theory about speed , as i think it may play into it. trinorthlighting may have a strong observation in play here.

[backdraft7]

But what about the speed of the pages

My pages are optimized for speed. That's usually step 1 in the webmaster process.

[Tallon]

no GA, our biggest site has been throttled for years

That's the most discouraging thing I've read in a long time. I've been struggling with throttling or a "something" for about a year now, with the hopes that whatever "this" is, it would pass.

My gut tells me that our root domain has been flagged somehow. A daily limit was assigned after team review. That root domain is "in the system" and balanced via tracking throughout the day.

This has been my gut feeling too, it's a handjob based on what I don't know? I suspect though it happens to sites that experience good growth and attract a lot of traffic.

What I Know:

--It's not from paid links or spammy inbounds (the site I see this on is 100% organic inbounds, no directory links, no article marketing, no guest posts, no paid links, nothing like that--though there could be some nefarious linking to the site by competitors that I can't find or see).

--It's not from selling links (I don't, never have).

--It's not from dup content, everything is original (other than quotes and links to those references).

--It's not because traffic is from search engines only (though mine is about 65%--from all search engines--, the rest is direct traffic, tens of thousands of subscribers, social media links and links from other sites).

What I do have:

--Google analytics installed.
--Use wordpress as a CMS.
--Monetization (display ads) other than adsense.
--Private whois

When the google traffic stalled:

--I was rising rapidly in the serps for both old and newly created content. Good growth month-after-month. A new article would typically sit somewhere in the top 10 of google. Now new content is maybe page two (if I'm lucky)...or 10, 15, 20 pages deep. Will rank for exact match or near exact match in title and is always indexed quickly.

--Old content (previous to Fall 2009) still ranks well and saw good month-over-month increases this spring/summer.

--New content (Fall 2009 and newer) is like it's sitting on a new domain.

It's like a "reset" button has been pushed for my site (old content fine and healthy, while new content trying to rank like a new, virgin site with no inbound links or authority would be).

What I don't see (btw I'm not an expert at digging in analytics and logs):

--Specific time throttling (pages only ranking at certain times of the day)

--Flat traffic levels day-by-day (I still have the normal fluctuations depending on the day of the week)

I have seen:

--Flat traffic levels month-by-month (from fall 2009 to spring 2010), there was a bounce over Spring/Summer 2010 from older content pages doing better. Trending down now for Fall, we'll see if it returns to the previous levels or if growth was allowed.

The WWW is well beyond what anyone could have predicted 20 years ago, and it grows every day in every way. Maybe, just maybe, "throttling" is Google's way of distributing traffic to sites that are pretty much equal.

If this is happening, then google's actually STIFLING the growth of the net. Think: Going Galt.

--Why create new content if it can't rank? If the cost of creating that content isn't justified, rate of new creation tanks. No matter who links to it, or how many links to it, it won't rank.

--If competitors and heavy hitters (for example demand media or mahalo) can take your ideas (from your new content), spin it enough so that it's original, and then THEY rank for it (and attract the links and new subscribers because they're ranking while you aren't)...why would you feed your competitors with new content ideas that you can't rank for?

I'm assuming most webmasters create content targeted for attracting and maintaining a specific visitor. If all you're doing by researching and creating perfectly targeted content is feeding your competitors with perfectly tuned content, and they can rank for it while you can't, and they grow from it while you can't...at some point don't you just say I can't afford to continue to help my competitors grow?

I've already cut my new content production rate by 20% this year just trying to ride out whatever it is that's going on, but now after a year of this, I'm wondering if my time and resources won't be better spent elsewhere or on developing other domains (that don't have throttling or "whatever it is" in place).

Maybe instead of developing one main domain for a brand, developing a network of domains around a brand is where the future is? That way when one domain gets "too big" and a switch is flipped for it to throttle its traffic, the others can still host and create new content that can still be used to gain awareness for your brand (and have a fair chance against competitors). It sucks because of the link dilution, but in the longer term, it might be a butt saver.

The most discouraging, dismal thing about all of this is that even if you follow google's guidelines for building and maintaining a site, they can (and seemingly do) stunt a site's growth, even if the only crime is "getting enough traffic from us already".

I think the answer is figuring out why it ISN'T happening to some of the giants on the net (ehow, mahalo, wikip, etc.). The answer to "fix" this, whatever it is, probably lies there.

[drall]

trinorthlighting, I will give that a shot.

Reno, as best as we can tell traffic is being sent to mashup type sites which all saw a large increase in search share starting at the same time as we all dropped. The traffic drop we can live with, though it's the first time we really have dropped in a large scale in 12 years it was just a matter of time before some update whacked us.

Comforting and disturbing to see hundreds of my competitors all going through the same thing. Funny thing is that according to quantcast Googles pageviews have skyrocketed right around the same time as Mayday rolled out.

Whitey, I see the same thing in most of our competitors. Search numbers look exactly like ours, no spikes, no curves just flatline except when mayday hit then they all dove at the same time and then reflatlined. None have recovered to pre mayday numbers. The site we are talking about serves pages at 1.1 - 1.3 seconds.

Tallon, Im sorry I got you discouraged. I think I first posted here about this in 05 or 06 but I cant remember. For us at least nothing has changed. I gave up and started putting my growth efforts other places.

I really think it all boils down to one thing, at the end of the day Google has a finite amount of unique visitors to spread around to us. As long as we play by the rules we are allowed an allowance of those visitors up to a point.

[trinorthlighting]

You can have te fastest server in the world and your pages might be optimized. But, google still has to test your page loading speed. From there the ago could be slapping some sort of calculations of daily visitors your site can handle. Thus the throttle occurs. If google serves up your visitors quicker than normal and is a bit behind in updating it's ago, that is where you might see your site top out traffic wise before the end of the day. There are a lot of calculations at play there, butt I can see the possibility.

Someone who has this issue should upgrade their server and see what happens.

Also, google could be shutting you down at certain times of the day due to communication hubs and general web traffic. Some hubs could top out at certain points of the day from google to your server thus slowing it down significantly enough for google to deem your site as slow and bam, your traffic goes to a faster site.

There can be a lot of factors here, but I suspect it has to do with page loading speeds at different times of the day.

[Whitey]

trinorthlighting - I'm going to keep an open mind on this as well.

I really think it all boils down to one thing, at the end of the day Google has a finite amount of unique visitors to spread around to us. As long as we play by the rules we are allowed an allowance of those visitors up to a point.

Could be , but I'd like to think not though. Wouldn't it make sense for Google to serve the most relevant sites.

Drall - one thing that strikes me is that you visitor count is substantial at 1M + visits . Could it be that your optimisation traffic driver techniques are at the "high end" with IBL's or are you purely content driven.

Have you isolated when / where your long tail phrases Yo Yo or show differently on different data sets ? - i mean if you are not getting traffic it must mean that at times your results are not showing.

[tedster]

Have you isolated when / where your long tail phrases Yo Yo or show differently on different data sets ? - i mean if you are not getting traffic it must mean that at times your results are not showing

Yes - that is the thing to discover. That and why it isn't happening in a more widespread fashion.

In the two cases I've been able to investigate (one a major enterprise site and one more entrepreneurial) both showed a lack of diversity in their backlinks. That is, it didn't look like either had many truly free editorial links, where someone just naturally decided to link to them. Neither seemed to have obvious guideline infractions either, but their backlink profiles were kind of lopsided.

That is only an observation, not a conclusion. There's no way two cases justifies a conclusion.

[Tallon]

I really think it all boils down to one thing, at the end of the day Google has a finite amount of unique visitors to spread around to us. As long as we play by the rules we are allowed an allowance of those visitors up to a point.

In a weird way, that's almost a relief to hear. If all sites in a niche were treated the same (allowed only so much traffic), I'd feel much better about it. Some are allowed much much more, why? Who knows. Maybe big, financed brands get a pass because they can afford fancy lawyers and google doesn't want the headache.

I'll sit and think hard til year end (watching stats) before making a final decision, but at this point I'm thinking it's time to "reset" things myself and look at pulling the plug to invest and dev. things elsewhere. Maybe do a 10% time/effort into the site to keep the site rolling minimally then the rest into a "sister" domain of some sort (that won't cannibalize the main domain's search traffic or links). It kinda sucks to pull the plug because I have a big pile of stuff waiting to upload into the site, but won't because of the reasons I mentioned previously. There's just no reason to hand feed it to my competitors.

[tedster]

Maybe big, financed brands get a pass because they can afford fancy lawyers and google doesn't want the headache.

I have seen pieces of this traffic behavior on at least one very big, well financed brand. However, it was for a very hotly contested keyword, and there are more than ten big brands who want to rank for it.

The implications of traffic throttling, if it is really happening as it seems to be, are immense. But I think all the guessing and theorizing won't get us very far. It hasn't in the many years that people have been noticing "something strange". And I don't have such a site under my care right now - or I would be digging like crazy to find out what the devil this phenomenon is.

If we can establish what is going on (and it might be more than one "thing") then we'll see where we can go with it. I'd particularly like to focus on the fact that traffic for some sites seems to be level across all entry pages - rather than just of one page or one keyword. If Google is intentionally creating such a limit or traffic cap, then that is a very big deal.

Several members have shared some of their data with us - and that's what we need if we're going to understand this.

[edited by: tedster at 3:55 am (utc) on Sep 21, 2010]

[backdraft7]

I really don't think "page speed" or a "limited number of visitors to go around" answers this current problem. I think Google is overwhelmed with data and the complexity is just breaking the system for the moment.

We see this same pattern at nearly every algo and system change. Google's Mayday, Caffeine and Instant all rolled out this year. I still think the best content makes it way to the top and I have a hard time believing that Google would unethically "throttle" sites. I hope their legal department would have the foresight to keep them from stepping into that quagmire.

Whatever they are doing, keep in mind, the web is still the "Wild West" of business and that all good things eventually come to an end. Google doesn't owe us anything, but it is in their best interest to keep the webmasters of the world in business.

The web should remain free, but you know it won't. My goal is to NOT keep putting all my eggs in one basket and to diversify my business so I don't have to rely on eventually being "fired" by Google.

I hope it straightens out in the next week or two, but again, I'm not holding my breath. Good Luck to all! Hang in there!

[Whitey]

Drall's situation looks interesting , especially with the cap on google referals.

But I'm concerned that there may be some confusion with issues that relate to a different type of Yo Yo where Google is confused or on the cusp of dishing out a penalty. The principle areas with my limited knowledge of where the Yo Yo can occur is in separate geo related Google data sets and a lack of authority ( lop sided link profiles being one aspect ) - which international geo region the links come from can be another.

My sense is that whilst Google's technology is ultra smart , the logic for consistancy with throttling is not there , and if it was it would be more widespread. I could be wrong , but that's how i see it.

Me thinks this is related to authority - but Drall certainly challenges that assumption.

Drall , on a large site like that you must have a lot of URL's Y/N ? Are they all cached and ranking with good link juice , or does the link juice kind of die in some areas of the site?

Did the new pages index and rank well when you added them ?

[squash1]

Hi everyone, just wanted to add that i sent Tedster the stats for my site the other day to show him the huge traffic spike that lasted for a few hours. One single page from my site constantly gets 4000-5000 visitors per day, then out of the blue, last thursday at 5pm it started getting 5000 PER HOUR which lasted till midnight. After this the traffic went back to its original number of visitors, which has stayed the same for the past 4 days. The ranking for this page has stayed in exactly the same position as far as i can see through all of this. It looks like google just opened up the taps for a short while and then closed them again. The keyword in question is a single keyword, no long tail, which gets around 9 million searches per month, and the site is an affiliate site. Considering i rank at position 3 for this keyword i would expect more than 4000-5000 visitors per day, more like the 5000 per hour that i got last thursday would be closer to the true amount this page would get. Also, the site is in the UK if that makes any difference at all. The site does not have google analytics on it either. I would really like to know what can be done get this higher level of traffic. Any ideas?

[dvduval]

drall, any relationship with the number of pages indexed? If google started to get a benchmark for what pages are worth in visits on average per day, it would be easy to throttle your traffic based on those benchmarks. It would not be a very complex computation.

[Jez123]

How can I test to see if my competitors are affected by this?

My site is nothing like any of yours by the sounds of it. Small niches with only 500 searches a day max (well that's the most I have got approximately).

[backdraft7]

@squash - are you sure there was no "news event" that might have prompted the inrush of traffic?
I find that the Google Trends tool can be quite telling...and I just discovered that one of my main two word key phrases seems to be on a major decline.

In addition I found that Google has added this useless "News for (keyword)" display to the top of the Web search page. If I want "News" I'll click the News link. The result of this is that my #1 position for a prominent 4 letter, single word keyword has pushed me down at least 5 positions, nearly beyond the fold.
This "News for (keyword)" display appears to be directly tuned into Twitter...as every few seconds it updates with tweets containing my target keyword.

Google Trends also shows that my sweet 4 letter one word keyword has also taken a deep six in the past few weeks. The bottom line is that much of the changes we are seeing might be the result of Google's adding stuff to push us off the page and promote Twitter and other social sites.

[firstconversion]

I've had two sites where I've seen this. I have no doubt it exists, but possibly only for some sites/sectors?

The major one was in the answer/tips sector. Flat line at 6000 visits per day for a year. Nothing we did moved this at all

Another in the books side, when we came out of the sandbox for two major terms after 6 months, we gained that traffic but lost an equal amount of long tail traffic, leaving us with exactly the same amount of traffic per day

[AlyssaS]

drall:

As with any good webmaster I have understood the power of a raw log file from the beginning I know what I am looking at and it is simply jaw dropping. The level of computational power Google must use to throttle boggles my mind.

Entire sets of randomly rotating serps that are tailored throughout the day to throttle us.

Like you say, they would have to use stupendous resources to traffic throttle you - and you need to ask yourself why they would go to that expense? What's the payoff from their point of view?

The real answer might be a lot simpler - they are rotating databases (perhaps they need to take one out while working on another, or perhaps the load at certain times of the day is such that they need to introduce another one). If the rankings on these databases are out of sync - say you are at #1 on one, but #9 on another, that would account for the fluctuations in traffic. (especially if they've introduced a database from another country - you may have been at #1 in the USA, but #9 in Germany for reasons of relevancy, and the German database may have been introduced to take the load of the American searches at certain points of the day)

There is an easy way to prove or disprove this theory: check your cache dates at various times of the day. If the cache date remains the same throughout the day, but your traffic gets dialled down during part of the day, then they are throttling your traffic. If the cache dates vary along with your traffic, you know they are simply rotating databases.

FWIW, from my observations, cache dates are all over the place. On Monday it may say that it was cached a day ago, and on Tuesday, it reports that it was cached a week ago - which makes no sense unless they rotated to a more out-of-date database.