Google Outages May 14

[sgietz]

For the past 30 minutes every Google service has been down, or sluggish beyond belief.

What the heck is going on? I've never seen anything like it, at least not regarding the big G.

[cmendla]

< moved from another location >

It is 1145 am eastern and I can't seem to get into anything from google. The pages are just hanging.

I also tried from my mobile phone (from the same carrier as my home internet) and google seems to be hanging there also.. Search, adwords, analytics etc.

Just wondering if anyone else was having issues.

[edited by: Robert_Charlton at 6:28 pm (utc) on May 14, 2009]

[bwnbwn]

< moved from another location >

In Texas I am having some big time loading issues with google right now. Is this just our area or are others having loading issues as well?

[edited by: Robert_Charlton at 6:29 pm (utc) on May 14, 2009]

[patrickSMC]

< moved from another location >

Many or most Google products seem to be down right now. I conjecture that these problems are related to the new features they are trying to roll out.

We had to temporarily disable analytics, because our site is getting hung up trying to request data from Google.

What else are people noticing?

[edited by: Robert_Charlton at 6:30 pm (utc) on May 14, 2009]

[tangor]

Another report from The Register:

[theregister.co.uk...]

[avalon37]

Yep that was quite inconvenient. Let me guess no email out to advertisers acknowledging this and thanking us for our patience?

[bwnbwn]

I kinda have a gut feeling it was a worker cut a main line and their recovery plan took howerver long to reroute.

Be intresting to see what it was as I as sure we will get someting from GooglePlex

[pontifex]

Lets say they added some of the new features on a broader base, so everyone sees either "rich snippets" or now really a map for more languages.

Lets say the average additional data of that information causes 10k additional traffic per query (map, stars, text) - that would be (with estimated 600 million queries daily) an additional overall (data) traffic of roughly 5,700 additional Terabytes or 5.7 Petabytes per day transfered. Lets say half of that lands on the US datacenters. That would be 3 Petabytes ADDITONAL traffic per day in the US. Now lets estimate around 60% happens in peak hours and that there are like 5 of those: mid day and early at night - that would be an additional 300 Gigabytes per hour additional traffic - carefully guessed.

Means: their SERPS changes cause (without the usual traffic) an add-on of around 12.5 Gigabytes per second or roughly 100 Gigabits per second additional traffic.

Now the DE-CIX, which is one of the 3 largest traffic exchange nodes in Europe pumps currently:

[de-cix.de...]

300 Gigabits per second for 70% of all European providers.

I wonder what will happen, if this sudden 30% increase from one source hits the web! Maybe an outage like described?

Any errors in the calc, folks?

P!

[randle]

They were pretty much totally down in the US Northeast for almost an hour.

Google in my life, hmmmm let me count the ways;

Gmail – no e-mail as we are running Gmail through Thunderbird
Adwords – Good chunk of our online paid advertising dead in the water
Analytics – All pages on lots of sites hanging up when loading
A/B testing through analytics on some sites, that wasn’t helping
Google hosted javascript library file – using that on a new interface we were testing out

Wow was that a wake up call on how overly dependent we have become on them. I guess nothings really for free.

[tangor]

how overly dependent we have become on them

And that dependency appears to be based on their suggested methods of search FOR THEIR PRODUCT. This slice and dice approach boggles the mind at times. What's wrong with just INDEXING THE WORDS?

<stump rant off>

[bwnbwn]

The report was there was an internal routing error and all the traffic was being routed through Asia and only effected 20% of the google users.

[pcmag.com...]

[skibum]

Didn't have any issues in PA, USA. Will they issue credits to companies for lost sales if analytics goes down and causes a site to hang, thus preventing transactions from taking place?

[WiseWebDude]

Google did apologize: [googleblog.blogspot.com ]

[stajer]

if analytics goes down and causes a site to hang

If you have the GA code properly integrated into your html, a browser won't hang displaying your page even if google.com is unreachable. Your page will render fully and immediately - though the js will not complete.

[Seb7]

BBC: Google apology for slow service.
[news.bbc.co.uk...]

ZDNet: Googlefail due to Asiatic misroute, says company.
[community.zdnet.co.uk...]

The Telegraph: Google crash hits millions of internet users.
[telegraph.co.uk...]

Reuters: Google disruption sets Web users atwitter.
[blogs.reuters.com...]

[edited by: Seb7 at 8:14 am (utc) on May 15, 2009]

[2clean]

I did notice this yesterday logging into Adsense

"On 16/05/09, our engineers will be performing site maintenance, from approximately 10:00 to 14:00 PDT."

So, I guess it's gonna happen again tomorrow.

One of the things I do find a little discouraging is that fact that this message does not attempt to address any publisher concerns that this message may bring.

IE, will the fact that the system is going down mean that for 4 hours my Adsense will not collect revenues.

2Clean.

ps: Just read the Official Google blog reply - Who writes this stuff? I feel sorry for the author who clearly believes that this is clear communication, rather than the condescending tripe it is. Online business managers have moved on from cuddly Google, do a brand review guys and speak with your audiences better.