sync our database with braintree #3473

chadwhitacre · 2015-05-21T18:40:35Z

Closes #3287. Braintree has received data from Balanced. Now we need to knit the customer records in Braintree with our participants table.

rohitpaulk · 2015-05-21T18:46:03Z

sql/branch.sql

@@ -0,0 +1,5 @@
+BEGIN;
+
+    ALTER TABLE participant ADD COLUMN braintree_id text NOT NULL DEFAULT '';


We have a braintree_customer_id field already, is that what you were looking for?

Ah, cool. :-)

#3473 (comment)

chadwhitacre · 2015-05-28T22:31:42Z

I'm seeing a handful of cases where an absorbed account has a balanced_customer_href, and the account that absorbed it has the same one.

chadwhitacre · 2015-05-28T22:35:32Z

I also see the seven customers that Braintree knows about that we don't.

rohitpaulk · 2015-05-28T22:45:15Z

bin/branch.py

+              , ("/customers/{}".format(customer_id),)
+               )
+
+    er = ExchangeRoute.insert(p, 'braintree-cc', card_id)


Before we do this, we should check whether a route exists for braintree-cc. If so, the user had updated his card, and we should just leave the participant as is.

chadwhitacre · 2015-05-28T22:49:27Z

We want a clean slate with Braintree going forward. We want exactly one Braintree customer per open account. I'm not sure yet about how cards should sync up.

rohitpaulk · 2015-05-28T22:55:06Z

We want exactly one Braintree customer per open account.

That's something I can handle next week as cleanup. As of now, we should be focusing on linking whatever braintree_customer_ids we have for people with valid cards.

rohitpaulk · 2015-05-28T22:58:32Z

One case where I see us having two braintree customers for one open account is #3473 (comment). It's safe to delete the 'migrated' customer.

chadwhitacre · 2015-05-28T23:00:03Z

I'm seeing a number of closed and a few suspicious accounts with a balanced_customer_href that matches customer_id in the Braintree import log.

chadwhitacre · 2015-05-28T23:41:46Z

I also want to see which cards are status: removed at Balanced. :-(

chadwhitacre · 2015-05-28T23:43:58Z

It would be Bad™ if we started charging users on cards that they had disconnected. Kind of a wtf to even have those in the export from Balanced, or at least not being invalidated at Braintree, if indeed that is what is happening.

chadwhitacre · 2015-05-28T23:52:34Z

Results using 8c34500 with the import log and the production database:

status	participants
ready	4,201
closed	247
suspicious	55
archived	48
already migrated	9
unknown	7
failed	0
total	4,567

chadwhitacre · 2015-05-28T23:54:32Z

So now for each card_id in the ready set, we want to find out the status of that card on Balanced.

chadwhitacre · 2015-05-29T00:14:29Z

I'm not seeing status in the Balanced API, either the docs or live.

chadwhitacre · 2015-05-29T00:17:22Z

The closest I'm seeing is can_{credit,debit}.

rohitpaulk · 2015-05-29T00:21:38Z

Aren't cards deleted entirely when we call unstore on them?

chadwhitacre · 2015-05-29T00:28:01Z

    status: function() {
        return this.get('can_debit') ? 'active' : 'removed';
    }.property('can_debit')

src

chadwhitacre · 2015-05-29T00:28:45Z

Aren't cards deleted entirely when we call unstore on them?

Is unstore a Balanced or Braintree API? I don't remember it from Balanced.

chadwhitacre · 2015-05-29T00:29:52Z

From #3391 (comment):

Spot-checking, I'm seeing a card that is "status: removed" at Balanced that is merely "Expired" at Braintree. 😮

https://dashboard.balancedpayments.com/#/marketplaces/MP12Xw5lL6iaILtqImIoroDL/cards/CC9ZBPfVyWiXwRNKrwSaO4m

https://www.braintreegateway.com/merchants/msjyyvtsvvnc4ny6/payment_methods/any/CC9ZBPfVyWiXwRNKrwSaO4m

chadwhitacre · 2015-05-29T05:20:13Z

I just satisfied myself that the SQL in get_participant_from_card_id does work.

rohitpaulk · 2015-05-29T05:23:03Z

🚀

chadwhitacre · 2015-05-29T05:26:23Z

👀

chadwhitacre · 2015-05-29T05:27:08Z

I'm testing a single sync. Then I'm ready to run it for real!

chadwhitacre · 2015-05-29T05:31:44Z

Okay! I'm seeing participant_id set properly at Braintree, and the braintree_customer_id and exchange route in Gratipay. Ready for lift off ...

chadwhitacre · 2015-05-29T05:32:57Z

Last chance for review ...

chadwhitacre · 2015-05-29T05:33:34Z

/me waits for Travis on principle ...

chadwhitacre · 2015-05-29T05:35:27Z

I'm expecting this script to run for 15 or 20 minutes, since there's more network traffic with the additional API and database calls.

chadwhitacre · 2015-05-29T05:35:54Z

Okay! 🚦

chadwhitacre · 2015-05-29T05:37:04Z

Script started.

chadwhitacre · 2015-05-29T05:37:54Z

Traceback (most recent call last):
  File "/Users/whit537/personal/gratipay/gratipay.com/gratipay/billing/payday.py", line 44, in g
    return func(*a, **kw)
  File "bin/sync-with-braintree.py", line 120, in process_record
    braintree.PaymentMethod.delete(card_id)
  File "/Users/whit537/personal/gratipay/gratipay.com/env/lib/python2.7/site-packages/braintree/payment_method
    return Configuration.gateway().payment_method.delete(payment_method_token)
  File "/Users/whit537/personal/gratipay/gratipay.com/env/lib/python2.7/site-packages/braintree/payment_method
    self.config.http().delete("/payment_methods/any/" + payment_method_token)
  File "/Users/whit537/personal/gratipay/gratipay.com/env/lib/python2.7/site-packages/braintree/util/http.py",
    return self.__http_do("DELETE", path)
  File "/Users/whit537/personal/gratipay/gratipay.com/env/lib/python2.7/site-packages/braintree/util/http.py",
    Http.raise_exception_from_status(status)
  File "/Users/whit537/personal/gratipay/gratipay.com/env/lib/python2.7/site-packages/braintree/util/http.py",
    raise NotFoundError()
NotFoundError

rohitpaulk · 2015-05-29T05:41:38Z

Hmm, only delete if the card exists? We'll also have to patch up all the other inserts/deletes to not repeat themselves if this wasn't the first record or so.

chadwhitacre · 2015-05-29T05:42:43Z

@rohitpaulk There was only one sync but a few deletes. Fortunately deletes are easier to work around.

chadwhitacre · 2015-05-29T05:42:49Z

Script restarted ...

chadwhitacre · 2015-05-29T05:45:00Z

At the end of this script, whit537 should have 0 cards attached at Braintree, though the customer should still exist.

chadwhitacre · 2015-05-29T05:49:17Z

Down from 26 to 12 so far. :-)

chadwhitacre · 2015-05-29T05:49:39Z

💇

chadwhitacre · 2015-05-29T05:59:00Z

2 left!

chadwhitacre · 2015-05-29T06:04:58Z

Okaaaaaay, more than 20 minutes. 🕝

chadwhitacre · 2015-05-29T06:05:37Z

No more 💳 for whit537.

chadwhitacre · 2015-05-29T06:06:49Z

real    25m32.430s
user    5m18.790s
sys     0m33.644s

chadwhitacre · 2015-05-29T06:07:57Z

So now a Customers & Payment Methods export from Braintree should have 3,619 records, right?

chadwhitacre · 2015-05-29T06:13:53Z

Sigh. 3,618. :-/

Oh well, close enough!

!m @rohitpaulk

rohitpaulk · 2015-05-29T06:14:12Z

!m @whit537

chadwhitacre · 2015-05-29T06:22:06Z

Log uploaded in https://github.com/gratipay/logs/commit/b9860556d64c5b8980494267456e8efc8478e451.

Start files for syncing with braintree

b20ecb2

chadwhitacre added this to the Balanced shutdown milestone May 21, 2015

rohitpaulk reviewed May 21, 2015
View reviewed changes

chadwhitacre added 2 commits May 21, 2015 14:50

We already have braintree_customer_id

5300db6

#3473 (comment)

Shelve work on migration script

dd0c890

This was referenced May 27, 2015

Radar 8 gratipay/inside.gratipay.com#209

Closed

run Gratipay 156 #3486

Closed

First draft of sync script

f32b1ed

rohitpaulk reviewed May 28, 2015
View reviewed changes

chadwhitacre added 4 commits May 28, 2015 19:19

Update script to skip some participants

67c3b34

pyflakes

0b5a4a6

Free up stdin for heroku config

eb22015

Tweak the logic a bit

8c34500

All systems go!

4c02e13

chadwhitacre added the Review label May 29, 2015

Work around non-idempotency

4481b8d

chadwhitacre closed this May 29, 2015

chadwhitacre deleted the sync-braintree branch May 29, 2015 06:13

		@@ -0,0 +1,5 @@
		BEGIN;

		ALTER TABLE participant ADD COLUMN braintree_id text NOT NULL DEFAULT '';

sync our database with braintree #3473

sync our database with braintree #3473

Conversation

chadwhitacre commented May 21, 2015

rohitpaulk May 21, 2015

Choose a reason for hiding this comment

chadwhitacre May 21, 2015

Choose a reason for hiding this comment

chadwhitacre commented May 28, 2015

chadwhitacre commented May 28, 2015

rohitpaulk May 28, 2015

Choose a reason for hiding this comment

chadwhitacre May 28, 2015

Choose a reason for hiding this comment

chadwhitacre commented May 28, 2015

rohitpaulk commented May 28, 2015

rohitpaulk commented May 28, 2015

chadwhitacre commented May 28, 2015

chadwhitacre commented May 28, 2015

chadwhitacre commented May 28, 2015

chadwhitacre commented May 28, 2015

chadwhitacre commented May 28, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

rohitpaulk commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

rohitpaulk commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

rohitpaulk commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

chadwhitacre commented May 29, 2015

rohitpaulk commented May 29, 2015

chadwhitacre commented May 29, 2015