PayPal Suffers First Major Outage

PayPal closed for maintenance yesterday.

[The Contractor]

PayPal was down for a while yesterday, but before they were they had problems.

I had 3 orders/payments yesterday totaling about $350 for that they never kicked an email out for (yes the email address works/fine).

Now they have a problem with their refresh on the withdraw form. If you select a different bank account than the first in the drop-down list, it will always refresh back to the first. Tried in both IE and Netscape 7.1.

Contacted them about the issues and of course have not heard back. Just sounding off is all...

Also, I have a merchant account etc. just the purchasers of these particular item(s) chose PayPal over my other payment options 90% of the time...

[woop01]

I still laugh at the bias in the headlines on this site. If it's a company the owner likes the title is "Extremely dependable service offline for the first time in five years". If it's a company not on the favorites list it's "Millions of dollars in payments fail due to problems with payment processor."

From the description for this thread...

PayPal was offline most of Monday.

Payments have been having problems since last Thursday and most users still cannot login to Paypal. Customers who click on the "Buy Now" or "Subscribe" buttons are still sent to pages that lag for 20+ seconds and I can't imagine how long an actual payment would take.

If I read the word 'Intermittent' to describe this one more time I think I'm going to puke. We can only HOPE this problem can be fixed by going back to an old version of the site and that Paypal will have the ability to go back to it quickly.

[grelmar]

I'm still wondering if it's DDoS or maybe a hack. If it was either of those, then PayPal would say it was because of a code upgrade. Better that, than admit to a breach in security.

[woop01]

If it was either of those, then PayPal would say it was because of a code upgrade. Better that, than admit to a breach in security.

Agreed. Any webmaster here would be up 24/7 working on this and would get it fixed quickly if it was their site. This is a company with a team of technicians who had planned this change for quite a while. It's getting really hard to buy the idea that this is a coding upgrade.

[FrankWeb]

I see some payments going through, but most of all lots of our clients who are complaining that their payments don't go through.

From the info received from PayPal I also do not think this was an update error. They gave the impression they were preparing the update very well. It sounds more like a hack or an attack.

The only thing that bothers me, is that since this is so huge, I wonder why PayPal does not issue an official statement or press release.

Or did they?

[sun818]

h**p://news.netcraft.com/archives/2004/10/12/redesign_cripples_paypal_service.html

[3dchase]

All I know is I hate the new design because now I have to look at that smug putz smiling at me on the log in page right before it goes to the your screwed page.

[sun818]

You can bookmark an internal page. Once you click on that shortcut, Paypal will ask you for your login-password and redirect you to the page you bookmarked.

[jchance]

It's getting really hard to buy the idea that this is a coding upgrade.

Agree. Any half way decent shop has a backout plan in place for any software migration... If it was a software glitch they would have backed it out long ago.

[oaktown]

I'm not one who believes in "black helicopters" or conspiracy theories, but I seem to recall that dmoz was slow and/or offline a couple of weeks back and around that same time, one of the ISPs I use went down (their site was down, they sites they hosted for myself and others were down, and remarkably all there phones were down (not busy) and their e-mail servers were down) all at the same time). Denial-of-service attack anyone?

[oaktown]

BTW, I can't get paypal to load at all just now.

[PizdusInc]

PayPal better get ready to pay out some serious $$$ to people... a lot of people are loosing serious money because of this outage!

[Hollywood]

PizduisInc - great point!

I contacted paypal.com and they refused to answer any questions regarding this sort of question, just kept getting the normal response... automated response from them.

They should have planned ahead for this, being a backend quality assurance type myself they could have avoided this for sure with proper planning.

Hollywood

[PayPalDave]

um... conspiracy theories and DoS attacks aside, let's apply Occam's razor folks -- that is, the simplest explanation is often the best / most likely to be correct.

evidence:
- PayPal launches new site / feature updates last Thu
- users experience slow response times / logins
- customer service + PR say "we know there's a problem, and we're working on it"

Occam conclusion:
- site upgrade caused the problem, & they're working on it
- not pretty, but not likely any black-hat conspiracies either.

PayPal has been doing upgrades every 6-8 weeks or so for about the past 2-3 years. most times there aren't too many blips; occasionally there are a few issues, and every once in a while they have a rough one that takes a few days to smooth out. seems like this is unfortunately one of the rougher ones, but i wouldn't presume any nefarious activity as the cause.

that said, i'm sure folks are busting their asses to get it fixed asap. no surprise there.

[sun818]

If we can reach the site, can you explain why graphics are not loading and why pings to www.paypal.com are not being replied to?

[edited by: sun818 at 6:22 pm (utc) on Oct. 12, 2004]

[blaze]

Are there any conclusions here? Unfortunately it's hard to test because you can't buy your own products.

Are people finding it fairly fixed or still ongoing?

[blaze]

One possibility is that a DOS attack was timed to hit paypal when they did their upgrade.

[woop01]

PayPalDave, a coding error can be backed out of by simply going back to the old version. A coding error would not cause simple images to take forever to load. How does a coding error for the website explain the fact that people couldn't use their Paypal debit cards for a period of time?

If a company that handles so much of our money and we trust with so much confidential information (sales figures, customer credit cards, OUR credit cards, our back accounts, our social security number) is hacked or compromised, what do you think the "PR" response would be?

I hope some state A-G forces Paypal to follow common banking regulations and soon.

[madmatt69]

I can't even connect to paypal.com - Not even to the home page. Constantly timing out.

And the sad thing is, even with people losing all sorts of money as a result, there's probably not much that can be done about it. I'm sure paypal has some sort of legal clause to protect themselves in the user agreements.

But I think needless to say, a lot of people are going to be looking at alternate services..

[Shannon Moore]

Looks like others tried to get some info on PayPal's issues by using the eBay "Live Help" feature -- right now when you launch eBay LH, you get the following long-winded note in the chat window:

"Welcome to eBay Live Help! We're here to help you get started.
We are aware that members are receiving intermittent errors when trying to login to or make a payment through PayPal. We are working to correct the problem as quickly as possible.
Please accept our apology for any inconvenience this may be causing."

Terrific for business all around, I should say! :(

[badtzmaru]

If users are continually refreshing pages in an attempt to access the paypal system, the load on the web servers is going to go through the roof. In that case, even static content will fail to retrieve. Once a site of this magnitude has an outage of more than a couple of hours it can become very difficult to bring the site back online - even after the original issue is fixed. Where the infrastructure may have been sufficient for maximum peak utilization plus a 25% safety margin, after an extended outage requests can come in at a rate one or two orders of magnitude higher. This much of an unexpected increase can cause a few not-quickly-correctable issues such as database or network infrastructure overloading.

I will agree that dedicating the proper resources and planning in advance could've prevented an outage of this length. However - it's also not anywhere as obvious to foresee or easy to remedy, as some are implying, when that planning and resource dedication is not devoted in advance and this situation occurs.

[blaze]

You make a good point often missed, but somewhere around 4am when traffic lightens up or so they should have gotten a handle on things.

[badtzmaru]

It seems that way, but no one is up trying to use it at 4am, so you have to consider that as a sort of "totally off" period. Once people start waking up, they will run to the computer to check paypal. Not just the people that usually check it on that day, but everyone who has had serious problems accessing the site for the last 4 or 5 days. So many people have been affected - everyone wants to get on, get their payments made, accepted, items shipped, and money out of there. And for sellers, if they are able to conduct their paypal business as normal, they are still going to start logging on and processing orders and transfering funds far more frequently than they did prior to the outage.

[sun818]

This is what happens eBay starts running things. They release sloppy, untested code and then have live users beta test the code for them. The Microsoft approach to development needs to stop.

[Shannon Moore]

Can anyone explain how a DDOS attack would affect ATM machines at all? People have reported problems with their Paypal debit cards at ATMS not working (it's mentioned here on Webmasterworld and also on the paypaldev forums).

Latest status from PayPal is posted: [www2.ebay.com...]

[sun818]

The use of the ATM machines is tied to Paypal.com in real-time so if you make any withdrawls it shows on Paypal.com right away. If the site is not working, it is reasonable to conclude this would affect all transaction types including ATM withdrawls or purchases.

[badtzmaru]

Sure. Given infinite traffic, a site will always have a first point of failure. Something has to be the first thing to reach its scalability limits, be it the web servers, app servers, DMZ network, bandwidth, or database systems. Certain of these systems would be shared across both the web and standard financial interfaces. A DDOS on the web side could cause internal network or database issues that could affect the ability of the system to process non-web financial transactions.

I am not implying that this is the paypal situation, just generically describing how a DDOS could adversely affect systems that are seemingly not directly linked to the web.

[Brett_Tabke]

Dave McClure leaves PayPal:

[500hats.blogspot.com...]

Friday was sort of my last day at PayPal... or at least it was my last day running the PayPal Developer Network program. PDN was the "baby" i birthed in 2001, and somehow managed to nurse along over the past three years into a pretty decent-size technical education site and community for some ~300,000 E-Commerce developers.

Totally coincidental is what the man says ;-)

He will be missed big time around there. Good luck Patrick.

[RedWolf]

PayPal seems to be a lot better. I can get in fine and poke around my account. The biggy will be if the backend is working though.

[sun818]

How about a free Paypal day - no fees to recipients for 24 hours?

[EquityMind]

Hmmm...hope that wasn't his parting shot....

Been real sporadic with me, can't even tell how much is in there....