analyze credit card failures

[chadwhitacre]

We have 140 bad credit cards in our database, out of 929 total. That's 15%! Ouch! :(

What's going on? What can we do to prevent failures in the future and recover from failures we're already seeing?

[chadwhitacre]

We get a short error message with each failure. Here's the breakdown for the past seven weeks:

$ ./analyze-failures.py 42

    26  Customer call bank
    23  no funding source
     7  Credit Floor
     5  Invalid CC Number
     4  Expired
     1  card was declined
     1  card number is incorrect
--------------------------------
    67  TOTAL

$ ./analyze-failures.py 43

    24  Customer call bank
    24  no funding source
     8  Credit Floor
     6  Invalid CC Number
     4  Expired
     1  card was declined
     1  Issuer Unavailable
     1  card number is incorrect
--------------------------------
    69  TOTAL

$ ./analyze-failures.py 44

    27  Customer call bank
    25  no funding source
    10  Credit Floor
     5  Invalid CC Number
     4  Expired
     1  card was declined
     1  CVV2/CVC2 Failure
     1  card number is incorrect
--------------------------------
    74  TOTAL

$ ./analyze-failures.py 45

    35  Customer call bank
    25  no funding source
    13  Credit Floor
     5  Invalid CC Number
     5  Expired
     1  card was declined
     1  CVV2/CVC2 Failure
     1  card number is incorrect
--------------------------------
    86  TOTAL

$ ./analyze-failures.py 46

    37  Customer call bank
    32  no funding source
    10  Credit Floor
     6  Invalid CC Number
     4  Expired
     1  card was declined
     1  CVV2/CVC2 Failure
     1  card number is incorrect
--------------------------------
    92  TOTAL

$ ./analyze-failures.py 47

    38  Customer call bank
    32  no funding source
     9  Credit Floor
     6  Invalid CC Number
     4  Expired
     1  card was declined
     1  CVV2/CVC2 Failure
--------------------------
    91  TOTAL

$ ./analyze-failures.py 48

    47  Customer call bank
    34  no funding source
    10  Credit Floor
     6  Invalid CC Number
     4  Expired
     1  expiration date is incorrect
     1  CVV2/CVC2 Failure
------------------------------------
   103  TOTAL

[chadwhitacre]

The four explanations that come to mind are:

• cards expire (or are turned off due to having been stolen elsewhere)
• card info is misentered
• Gittip charges are being flagged as fraud
• card balance is low

Can we infer anything from the above error messages about what scenarios we're hitting? Maybe someone from @balanced can comment on the meaning of the error messages?

Ping @mjallday @jkwade @mahmoudimus.

[chadwhitacre]

I emailed @balanced support about this as well.

[mjallday]

Hey Chad!

I had a quick look, it doesn't look like you're passing the country_code param for any cards (it does appear in the meta data but not the actual card body) and I wonder if that could be a possible issue for foreign cards.

Is it easy for you to break down the rejections by US/non-US so we can see if that's a factor?

The docs show how you can pass country_code.

[mjallday]

I had a quick look at your code, you're using an ISO2 country code, Balanced accepts an ISO3 :)

Essentially we could make a simple change like this

diff --git a/www/assets/%version/gittip.js b/www/assets/%version/gittip.js
index aa69ac4..4674e9f 100644
--- a/www/assets/%version/gittip.js
+++ b/www/assets/%version/gittip.js
@@ -392,7 +392,7 @@ Gittip.submitPaymentForm = function(e)
     }

     var credit_card = {};   // holds CC info
-
+    country = $('select[id="country"]').val();
     credit_card.card_number = val('card_number');
     if (credit_card.card_number.search('[*]') !== -1)
         credit_card.card_number = '';  // don't send if it's the **** version
@@ -400,7 +400,8 @@ Gittip.submitPaymentForm = function(e)
     credit_card.name = val('name');
     credit_card.street_address = val('address_1');
     credit_card.region = val('state');
-    country = $('select[id="country"]').val();
+    credit_card.country_code = country_code;
     credit_card.meta = { 'address_2': val('address_2')
                        , 'region': credit_card.region // workaround
                        , 'city_town': val('city_town')
________________________________________________________________________________

But we also need to switch the country codes in utils.py to ISO3 and I don't know what impact that would have on gittip itself. We could also extend it so that your form uses iso2 and 3.

[chadwhitacre]

Awesome, thanks @mjallday! The ISO2 codes were added by @chrisdev when he added the country field to the credit card form in the first place. I believe the credit card form (www/credit-card.html) is the only other place we'll need to update that.

[mjallday]

will it ruin any existing queries if i just go ahead and change it?

[chadwhitacre]

I'd say go ahead with a pull request if you're up for it. Thanks! :D

[mjallday]

@whit537 @msherry also reminded me that we tend to map some other response codes under the generic description. they should look like R000. do you have those available and can you split the generic group by that?

[chadwhitacre]

I ended up backing out the change on #919, see notes there. Need to spend more time with this.

[chadwhitacre]

@jacobian and @wraithan in IRC:

FYI, 15% delinquency rate isn't terrible. It's bad, but 10% is about average.

yeah, I've seen it at about 7-13% for the various ecommerce sites I've worked on

[chadwhitacre]

Got a follow-up text message from @jkwade on this:

Yo, Chad, is Gittip still having a lot of card declines. I've looked at your logs. There are some declines, but it seems most cards are being charged successfully.

I would like to dig into this but at the moment I'm not seeing it as bad enough to raise an IRQ.

[olivierlacan]

I had an expired card on Gittip and was never notified so it's been sitting there and I'm not sure what happened.

I assumed no donations failed because I still had a positive balance, but what would happen if my balance ran dry?

We preemptively email people when their credit card is about to expire at Code School, which reduces those issues somewhat.

[chadwhitacre]

@olivierlacan I've added a +1 to #583 for you. :-)

[chadwhitacre]

Andrew from Balanced support, over a Heroku cc failure today:

The error response here is for a generic "Banking" decline, which comes directly form the bank and unfortunately doesn't tell us anything. The only way to resolve this issue for this specific card is to have the cardholder call the bank.

That being said, in the past we've found that switching a marketplace onto their own specific merchant ID can help drastically cut down on this type of problem. I'm in the processing of moving you guys over as we speak, and this change should come in to effect in the next few hours (this doesn't require any changes whatsoever on your end, FYI).

Please let me know if you have any questions about this, or if there's anything else that I can help you with!

Awesome, thanks Andrew. Can you tell me more about the merchant id? Is this something that's exposed through the dashboard? It's an identifier with the bank on the bank end, I take it?

Yep, it's an identifier code that's tagged onto all credit card transactions that come from Gittip, but it's not something that is exposed on the dashboard (or that should effect your transactions in anyway other than cutting down on declines like this). Just incase you're interested, your new merchant category code (MCC) is 7299.

7299 is "Miscellaneous General Services."

http://usa.visa.com/download/corporate/resources/mcc_booklet.pdf
http://en.wikipedia.org/wiki/Merchant_category_code

[chadwhitacre]

In week 74 (#1632) we jumped 13% in credit card failures, which is a lot. I don't have all logs on my laptop, here's the data based on what I have:

$ for name in `ls *.log`; do printf $name; grep 'Charging.*failed' $name | wc -l; done
gittip-66.log     179
gittip-67.log      -
gittip-68.log     184
gittip-69.log     195
gittip-70.log     196
gittip-71.log      -
gittip-72.log     201
gittip-73.log     203
gittip-74.log     233

Ironically, Balanced is among those whose card failed this week.

[chadwhitacre]

Twitter

[chadwhitacre]

I've emailed Balanced support and it sounds like we'll debug here.

[jkwade]

Thanks for getting this data for us @whit537. I think this issue is related to the Merchant Category Code (MCC) change we put in effect last week, but don't have concrete evidence of that yet, just correlation.

[chadwhitacre]

Thanks @jkwade.

[jkwade]

@whit537 et al, We looked into the increased declines from today. Gittip 73 ran successfully with the new MCC, so I don't think that's the problem. Both Balanced and cardholders' banks (e.g. jdorfman's bank) have fraud filters that flag transactions that appear to be abnormal. Since Balanced has a strong partnership with Gittip, we've decided to dial down our internal fraud filters (don't ask us what they are that would defeat the purpose of having them). We'll continue to monitor and calibrate. Keep in mind, this does not mean that declines won't occur in the future, either from Balanced or the cardholders' banks.

Sorry for the inconvenience.

[jkwade]

Anyone that would like more insight into Balanced's fraud fighting philosophy might find this blog post interesting.

[chadwhitacre]

Thanks @jkwade. To date, the answer when we've asked about specific declines (such as @jdorfman's) has been that each decline has been at the bank layer. Without delving into specifics, do I hear you right that the spike in declines this week was due to fraud filters at the Balanced layer instead?

We did talk a year ago about increasing the data we send to Balanced for the purposes of fraud signaling (#362). That would probably help here, eh? In addition to @mjallday's suggestions above?

Apart from that, I suppose the immediate action item is to see whether those 233 declines drop back down towards 203 next week.

[chadwhitacre]

Leaving this as an IRQ until next week.

[ericholscher]

So what happens to people that were giving money and now no longer are? They aren't alerted in any way, and anyone who was receiving money from them just loses that?

[chadwhitacre]

@ericholscher There are no automated notifications, no. That's #583. I've manually notified both Heroku and Balanced that their card is failing. I will follow up next week to make an effort to ensure they participate next week. Feel free to do likewise. :-)

For the ~30 new failures this week due to Balanced's fraud filter, we're expecting those to not fail next week because Balanced has relaxed their internal fraud filter for us.

Are there other steps you can see that we should take here?

[mahmoudimus]

@whit537 I think @ericholscher's use case is very interesting. There might be an expectation of a certain ceiling of tips per week, so when there's a decrease in expected earnings per week happens, there could be anxiety, etc. @whit537 does gittip offer retries in case a tip fails? Going to watch this issue develop as a potential usecase for balanced/billy /cc @victorlin

[chadwhitacre]

@mahmoudimus Agreed. It kind of snuck up on me, but we now have at least three people (@ericholscher @alexpott @ashedryden) relying on Gittip as their primary source of income. That's a weighty responsibility for Gittip and we need to live up to it.

Gittip retries credit cards week by week indefinitely, but we don't immediately retry after a failure in any given week. I guess I've assumed that an immediate retry is highly unlikely to succeed. Is it?

[msherry]

@whit537 Yes, immediate retries are unlikely to succeed, and are likely to look even more fraudulent to the issuing bank. We recommend retrying no more than once every 24 hours.

[mahmoudimus]

@whit537 recurring billing has a ton of issues with what the payment industry calls, 'soft declines', which are just declinations that can be retried using a back-off window. So try failing cards 1 day later, then if that fails, try it 2 days later, then if that fails, try it 3 days later - so on and so forth. This is a specialization of offering [Dunning](http://en.wikipedia.org/wiki/Dunning_(process\)) support in a recurring billing system.

This will provide more analytics such as how much expected tips were not given per payday? or projected tips given velocity of increase / decrease in tips, etc.

[chadwhitacre]

Sounds like we're on the right track with the weekly retries, then. We could retry on Fridays but if that fails I wouldn't want to retry again on Monday or Tuesday, because then it gets into the next week and encroaches on the following payday. I've reticketed retrying on Fridays as #1637.

[chadwhitacre]

@mahmoudimus @jkwade Any chance you could re-enter the Balanced credit card on Gittip so you're back on the homepage and the numbers go back up? :-)

[wraithan]

A manual retry after being notified that it failed would also be nice. I
know notification isn't built. But if I found out later that my card was
declined, I'd like to change/check it and manually try again.

[chadwhitacre]

We've recovered somewhat but not entirely:

$ for name in `ls *.log`; do printf $name; grep 'Charging.*failed' $name | wc -l; done
gittip-72.log     201
gittip-73.log     203
gittip-74.log     233
gittip-75.log     222

[chadwhitacre]

Sorry, I should've opened a new ticket for the rash of cc failures two weeks ago (#915 (comment)). I'm removing IRQ from this ticket.

[chadwhitacre]

Setting to three stars, reticket specific action items.

[chadwhitacre]

Closing. If anything comes up again related to this, reticket.