Authorize.Net Down?

Authorize.net Timing out...

[Robski]

It appears that Authorize.Net is down and has been for about 5 hours.

I can't load [authorize.net...] nor [secure.authorize.net....]

Does anyone know if they have an offsite status page or what is happening?

[MattInFla]

They appear to have gone down just after midnight Pacific time this morning.

Unable to reach any of their domains, and so far the toll-free lines all fail to go through.

A call to their main office in Utah indicates that they are "closed for the holiday".

Here are the numbers I located from Google's cache:

Authorize.Net
808 East Utah Valley Drive
American Fork, UT 84003
Tel: [phone]
Fax: [phone]

10800 NE 8th St, Suite 600
Bellevue, WA 98004
Tel: [phone]
Fax: [phone]

CyberSource Corporation HQ
1295 Charleston Rd.
Mountain View, CA 94043
Tel: [phone]
Fax: [phone]

Not looking good at the moment. I'm just hoping this is a technical issue, and not the company folding up and leaving town in the middle of the night.....

[emilygems]

In the meantime, we are losing sales. I have had 6 calls this morning from customers trying to place orders. I sure hope not everyone at authorize.net is on holiday.

[diamondgrl]

I called to their British office. There is was a fire at the data center. That's the only thing they are able to say at this point.

This is not fun ...

[emilygems]

I am taking information manually from the customers who call and hoping this gets resolved so that I can later run the transactions...but I sure can't do this all weekend!

[maardsma]

Wow, pretty serious, and unprecedented in my 5 years of running e-commerce sites. What do you guys think about a contingency plan like getting switched over ASAP to a gateway that supports authorize.net emulation?

Also information about authorize.net's disaster recovery plans whether they have a backup data center would be helpful in assessing what kind of trouble we are in.

We can do voice authorizations by calling Visa, MC, etc right? Capturing those transactions later using capture only manually in authorize.net's terminal could be a bit of a nightmare though.

Let's discuss solutions.

Mark

[mblyman]

I am also seeing authorize.net down and people are not able to place orders on our website. I hope we can see authorize.net come online soon!

[emilygems]

I am placing a notice on my website for customers to use Paypal in the meantime...but I am afraid of scaring people by saying there is trouble with my payment gateway. I'll just have to word it nicely. But this is an immediate solution to recover some of the sales that are being lost.

[CernyM]

Other sites are reporting that there was a fire at their server center and they are awaiting permission from the fire department to get inside the building and initiate their backup plan.

[rickkai]

Our sites have been down since 2 am EDT this morning. We will direct folks to use Paypal and have phone reps take orders over the phone using another site that we have integrated into CyberSource (which works...). Anybody know of a quick way to switch to the CyberSource API?

[ssgumby]

We too are down .. I switched over to google checkout but it is flaky as it doesnt take my coupons properly. I guess its better than losing everything.

We are seeing if we can do an emergency switch-over to linkpoint

[trinorthlighting]

We flipped back on Google Check Out in the mean time. This is a good lesson to all to have a back up plan themselves ie... Google Check out or Pay Pal

[ssgumby]

Ok, this just in. Spoke with my merchant account rep and they say authorize.net will be back onlin by 1pm Eastern Time.

[WebDev2009]

Can anyone else confirm this?

[maardsma]

I am seeing network connections and pings to authorize.net addresses now that didn't earlier, so some progress seems to be happening. I am guessing they have routed their DNS to alternate addresses as part of their recovery plan. But I would not hold my breath for 1pm knowing how these things go in general.

[johnh123]

I can't believe they didn't have some kind of better disaster plan.

[ssgumby]

They (merchant rep) just told me they've been working for a few hours switching to offsite data center. This would explain others getting pings now as they brought the new servers up. I assume it will take hours to move the database over if they dont have a replication system in place.

The good news is it sounds like they have a plan. This kills us, but things happen out of our control all the time. How they respond s key.

We plan to start email/calling all abandoned orders once we are back.

[trinorthlighting]

I have heard 1pm as well. They are also working on their home page as well to get that online with information as well which should be up soon.

[robsnell]

[dslreports.com ]

Illustrating the continued fragility of the modern Internet, threads in both the Zen Cart and Web Hosting Talk forums indicate that one of the country's largest credit card authorization networks has gone completely offline. According to several posters, a "perfect storm" was created by the combination of a data center fire and the July 4 weekend holiday, resulting the company going completely dark for more than five hours and counting. The company's phone lines and this Authorize.net website remains offline as of this writing. If your weekend beer purchase didn't go through, now you know why...

[Gomvents]

this is a nightmare, I've lost over $1,000 during this downtime, nearly a hundred nasty emails to me... clients are furious that I recommended authorize.net.... not good! Credit card processing should be the most reliable thing, not like this!

[incrediBILL]

There doesn't seem to be much coverage on this in the press.

I found this article:
[ireport.com...]

Not much information except that there was a fire, nobody is answering the phones and probably won't until next Monday.

When it comes to ecommerce, that's a massive FAIL for any payment processing company.

For merchants only using a single payment processor without a backup plan, it's also a massive FAIL and I'm sure tens of thousands of merchants have just learned this lesson the hard way.

You should always have a backup plan like Google Checkout, even if it's Paypal, just in case you have a serious outage.

FWIW, many of you probably are unaware that depending on your payment processing installation method, the payment processing center is unavailable to your customer more often that not.

The solutions that transfer the customer to the payment processor in the browser are most prone to failure due to transient networking errors rampant on cable networks. The connections are more stable to your payment processor if you do a silent connection on the back end of the web site like I do which is direct data center to data center over a major network backbone.

[maardsma]

I switched my sites to save CC info and accept orders but not attempt to contact authorize.net. We will attempt to charge CCs after a gateway is back online (authorize.net or a competitor if this gets serious enough). So we'll only have to call customers if cards decline later. And we can do voice authorizations for anything that needs to ship today. -Mark

[ssgumby]

Keep track of this on the official twitter account /authorizenet

[edited by: engine at 5:50 pm (utc) on July 3, 2009]
[edit reason] TOS [/edit]

[Gomvents]

incrediBILL, We have PayPal as a back-up. Customers don't like to be told what to do. When I suggested to one customer that they could complete their order with PayPal they told me "or you could just make your credit card system work like a real company! You lost my order!". I am documenting all of this. They need to make this right and fix their systems to prevent this from ever happening again. They certainly make enough money to put the proper systems in place. In what are they thinking not being open 24/7 365?

[Aldebaran]

This from authorizene [twitter.com]t

"We are down due to a fire at our primary data center in Seattle. We are working to restore services, but no ETA at this time."

Jill

[edited by: jatar_k at 3:49 pm (utc) on July 3, 2009]

[edited by: engine at 5:53 pm (utc) on July 3, 2009]
[edit reason] authoritative links only thanks [/edit]

[ssgumby]

How can you prevent a fire from ever happening again? Disasters happen, its how they handle it thats important. If they are working on moving to their disaster recovery centers then that is perfect.

From what i've heard the fire wasnt even IN their center,it was in the same building and triggered sprinklers in their center. Total disaster from what ive read.

[kaz]

Dotster hosting has the same sort of thing going on.
Same fire?

[edited by: engine at 8:12 pm (utc) on July 3, 2009]
[edit reason] See TOS [/edit]

[incrediBILL]

Customers don't like to be told what to do.

You don't tell customers to use Paypal, your site needs to drive the process, the process can be seamlessly integrated into your site.

Alternatively, disconnect authorize.net and just store the orders now so you can manually process them later.

Here's a list on PayPal's site supposedly indicating which carts are integrated using their Website Payments Pro, perhaps yours is on the list!
[cms.paypal.com...]

[edited by: incrediBILL at 3:37 pm (utc) on July 3, 2009]

[trinorthlighting]

Gosh, if you are following their twitter page they are now saying their back up data center was impacted as well!

[ssgumby]

They say transaction processing is backup but we are still failing.