Welcome to WebmasterWorld Guest from 54.160.254.203

Forum Moderators: incrediBILL & martinibuster

Message Too Old, No Replies

Google meltdown?

     
11:29 pm on Sep 25, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Feb 12, 2005
posts:359
votes: 0


hopefully temporarily and hopefully only me but i can't access any of my web sites as they seem to be hanging on trying to open Google Ads
the adsense log in page isn't opening either

anyone else?

11:33 pm on Sept 25, 2006 (gmt 0)

Preferred Member

5+ Year Member

joined:June 10, 2006
posts:606
votes: 0


I just checked mine to see. The ads loaded a bit slower than normal, but they opened up as always.
4:34 am on Sept 26, 2006 (gmt 0)

New User

10+ Year Member

joined:Feb 13, 2006
posts:24
votes: 0


Are you using a Unic/Cpanel Server? Does the page load fine when using Firefox?

There is a known exploit that occured in the last few weeks and Cpanel has issued a fix - and not all servers/datacenters have gotten around to applying it.

The funny thing about it - it only shows itself when using IE - and Google Ads definitely make it worse. On pages that do not have Google ADs or that type of code - the problem is stil there, but almost un-noticable.

When using Firefox - the problem goes away...

READ FROM HOSTING RELEASE:

This was a 0 day cpanel exploit. Anyone in the world running cpanel could have been exploited.

They actually did the cpanel exploit about a month ago which explains what we thought at the time to be "bad cpanel updates." We thought this because sites weren't loading in IE and the fix was just changing a line in cpanel. At this point in time viruses weren't loading as far as we knew or heard so there was nothing to suggest anything different then a bad cpanel update.

We believe whoever did this was perfecting what they were about to launch and waiting for the right moment. They chose a few days ago to launch it in full force to exploit Microsoft's newly announced vml exploit. They used the exploit in cpanel to distribute trojans / viruses to target the vml exploit.

Here's what cpanel said once we showed them the exploit....

Quote:
Originally Posted by Cpanel
"This has been confirmed and patched. Running /scripts/upcp will fix the vulnerability in all builds."

We had our own patch we ran before the release of cpanel's, and as soon cpanel provided an update we ran their patch as well.

We had a few problems to figure out......

1. How was it happening and where was it coming from?

We could easily fix the problem but every time we did in minutes to an hour later it would come back. After hours of looking how this was happening we made little / no progress. We reached out to the web hosting community for help and soon had everyone helping us.

To name a few....

ThePlanet's security team, Layeredtech's security team, idefense.com, verisign, our best inhouse administrators and gurus, some server admin companies, and a customer of our's named Brad who helped build the architecture of paypal and ebay.com.

Brad had some contacts in symantec, trend, and Mcafee that he was able to contact on our behalf. We had everybody working on this. Our CTO DaveC finally figured out what was causing it and cleaned it up at which point it has not happened since.

2. What was exploited and how?

We might have cleaned it up to fix the problem, but without knowing how they were exploiting our boxes they could easily do it again and again. One of our best admins Tim Greer solved this mystery today when he came across a cpanel root exploit that nobody knew about. He tested it and it was soon proven this cpanel root exploit is how the hackers had the power to do the redirects. As soon as we knew the function of cpanel that was being exploited we had help with the creation of a bandaid patch that was applied immediately.

At the same time this was going on I got on the phone calling everybody in the industry to get cpanel involved. I was able to reach cpanel's operations manager Dave who quickly came up with a patch that has now been released to the public. At this point we ran upcp which will prevent our boxes from being exploited this way again.

12:47 pm on Sept 26, 2006 (gmt 0)

Junior Member

5+ Year Member

joined:Aug 2, 2006
posts:69
votes: 0


if you use google analytics, the code has been creating problems in loading pages. I tested it on various pages and it definitely was google analytics being the cilprit.

seemed ok after a few hours though.

1:53 pm on Sept 26, 2006 (gmt 0)

New User

10+ Year Member

joined:June 13, 2005
posts:4
votes: 0


Unable to login into adsense account, ads not showing up on site either.
What gives?
2:08 pm on Sept 26, 2006 (gmt 0)

Junior Member from US 

10+ Year Member

joined:May 18, 2005
posts:165
votes: 0


Same here. No ads showing up on any sites I regularly visit which use Google ads. Must be due to Payment in Progress(?)
2:55 pm on Sept 26, 2006 (gmt 0)

New User

10+ Year Member

joined:Mar 30, 2004
posts:37
votes: 0


Everything is not working for me as well..no gmail, no search engine, no adsense ads...

~shiva

3:07 pm on Sept 26, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Sept 21, 2000
posts:69
votes: 0


Same here as well, Can't get to google search, adsense, no adsense ads or adwords login. I was able to get to each once this morning about 20 minutes ago after an hour of nothing, now back to being unable to access anything.

It seems to be very sporadic and dependant on location as 2 of our other offices nearby on differnet networks are seeing everything normally.. When I was able to log in to adsense, impressions and CTR's seemed fairly normal for this time of day.

[edited by: Shadoze at 3:08 pm (utc) on Sep. 26, 2006]

3:27 pm on Sept 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Hi

Fine to me from the UK via a couple of different routes.

Is someone taking a pop (DDoS attack) at G again?

Rgds

Damon

3:31 pm on Sept 26, 2006 (gmt 0)

New User

10+ Year Member

joined:July 18, 2005
posts:7
votes: 0


I'm in the NY area... Can't even access google.com

Steve

3:46 pm on Sept 26, 2006 (gmt 0)

Junior Member

5+ Year Member

joined:Mar 28, 2006
posts:172
votes: 0


Weird - in UK, and all appears fine.
3:48 pm on Sept 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member sem4u is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Dec 18, 2002
posts:3061
votes: 0


All working fine from the UK.
3:52 pm on Sept 26, 2006 (gmt 0)

New User

5+ Year Member

joined:Apr 2, 2006
posts:23
votes: 0


Ads not appearing on my sites. (Balt/DC)
4:10 pm on Sept 26, 2006 (gmt 0)

Preferred Member

5+ Year Member

joined:June 5, 2006
posts:352
votes: 0


In Boston area, no way to access google or gmail or adsense or even website that have adsense ads. Will keep updated. Noticed it first thing in the morning here.
4:20 pm on Sept 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Hi,

Well, I seem to even be getting a very reasonable amount of click-throughs on AS, so maybe this is a US-based-user-only problem...

Back to the earlier controversial "Is there anything worthwhile outside the US" thread... Today, yes, ONLY outside the US! %v<

Rgds

Damon

4:30 pm on Sept 26, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Nov 28, 2004
posts:566
votes: 0


Fine all day in Spain
5:27 pm on Sept 26, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Aug 8, 2005
posts:453
votes: 0


Oddly, I'm just a short distance north of Boston/New York and have had no problems all day.

The Internet Traffic Report is showing some problems in Maryland, Florida and Texas which may be causing issues for users working through those networks.

Chapman

6:12 pm on Sept 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Hi,

I'm *possibly* seeing a slightly lower volume of US-user AS clickthroughs relative to UK-users today.

Rgds

Damon

7:03 pm on Sept 26, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:May 19, 2004
posts:174
votes: 0


Europe, Flanders
Everything works exept login int adsense and adwords.
Gmail=ok
Ads are on my sites
search works
7:47 pm on Sept 26, 2006 (gmt 0)

Junior Member

10+ Year Member

joined:Mar 14, 2003
posts:198
votes: 0


I'm having all kinds of problems with ads not appearing, ads mismatched to content and ads that generate a Norton intrusion warning (see separate post about that).

Google must be having a bad server day or something.

7:52 pm on Sept 26, 2006 (gmt 0)

Junior Member

5+ Year Member

joined:Sept 10, 2006
posts:98
votes: 0


Nothing seems to be working. What's going on? Ads are still displayed on my site but some parts of google.com don't work.
7:59 pm on Sept 26, 2006 (gmt 0)

New User

10+ Year Member

joined:Oct 26, 2004
posts:10
votes: 0


We have a store that sees alot of daily sales from adwords. Sales are down the pan today and analytics is producing zero data. Google.com is appearing fine but as for the rest.. I'm in our Vancouver, BC office.
8:00 pm on Sept 26, 2006 (gmt 0)

Preferred Member

10+ Year Member

joined:Mar 1, 2004
posts:433
votes: 3


No problems here (US).
8:30 pm on Sept 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Hi,

Google Analytics has not updated for me for twelve hours, which is not unprecedented, but is unusual.

I suspect that G has turned off all non-essential processing until it can get on top of the current problem.

I noticed yesterday that pack.google.com was just taking my browsers round and round in a loop: I wonder if that is in any way connected? If not, I'm guessing a DDoS attack.

Rgds

Damon

8:36 pm on Sept 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 1, 2004
posts:1987
votes: 0


OK here in maryland usa...

Well.. except for the foot-thick coating of sky all over my lawn.

9:12 pm on Sept 26, 2006 (gmt 0)

Preferred Member

5+ Year Member

joined:June 5, 2006
posts:352
votes: 0


Everything returned to normal around 5 PM Eastern (USA).
9:20 pm on Sept 26, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Sept 25, 2004
posts:2156
votes: 0


Hi,

Yes, data has just started appearing in GA again for me, though with a huge chunk missing for now.

Will the code-monkey you put the infinite loop into G's brain please take a bow? %^P

Rgds

Damon