run Gratipay 247

[gratipay-bot]

← Payday 246

---

Docs

http://inside.gratipay.com/howto/run-payday

Checklist

• Review accounts
  • New givers
  • 1.0 Payout applications
• Trigger backup
• Run payday script
• Commit payday logs
• Run masspay
  • DB query to update broken paypal fees
  • Generate paypal inputs
  • Verify there is no fee mismatch
  • Submit masspay on paypal
  • Post back masspay
• Commit masspay logs
• Review decline rate
• Shuffle escrow

Rotation

Month	Pilot	Copilot
March	@whit537 ✈️	@clone1018
April	@rohitpaulk	@whit537
May	@clone1018	@rohitpaulk
June	@whit537	@clone1018
July	@rohitpaulk	@whit537
August	@clone1018	@rohitpaulk
September	@whit537	@clone1018
October	@rohitpaulk	@whit537
November	@clone1018	@rohitpaulk
December	@whit537	@clone1018
January	@rohitpaulk	@whit537
February	@clone1018	@rohitpaulk

[clone1018]

Rebuilding a VM to run payday on

[clone1018]

Ok, all setup with a working Gratipay. I transferred all the logs and backups from the old server.

[clone1018]

I'll run payday a bit later.

[clone1018]

Backup the Gratipay database to ../backups/2017-02-23.psql? (y/N) y

[clone1018]

I've never ran a payday without SSL warnings :)

UNTIL NOW

[clone1018]

(MainThread) Script ran for 40 seconds (0:00:40.663624).

[clone1018]

gratipay::MAROON=> update exchange_routes set fee_cap = 20.00 where fee_cap is null and network = 'paypal';
UPDATE 9

[clone1018]

I can postback!

[clone1018]

POSTed MassPay back to Gratipay for 62 users.

[clone1018]

[clone1018]

Escrow Shuffle

	$
MassPay	1,750.47
5x	8,752.35
Balance	8,336.72
Added	0.00

Gonna wait until next week

[clone1018]

Paydays is done!

[clone1018]

Updated ✈️

[chadwhitacre]

I'm not seeing that I received an email notification this week. Investigating ...

[chadwhitacre]

Not in spam.

[chadwhitacre]

Many fewer sends this week.

[chadwhitacre]

I'm finding log messages in the SQS queue. I'm going to see about downloading those ...

[chadwhitacre]

Here's the script I used. I created temporary credentials for my user account in IAM and set those as the default in ~/.aws/credentials.

[gratipay] $ run defaults.env local.env ./download-email-status-queue.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import json
import sys

import boto3

sqs = boto3.resource('sqs')
queue = sqs.Queue('https://sqs.us-west-2.amazonaws.com/482083521290/email-status-notifications')

nempty = 0
while nempty < 10:
    nprocessed = 0
    for message in queue.receive_messages():
        try:
            msg = json.loads(message.body)
            body = message.body.encode('utf8')
        except:
            import pdb; pdb.set_trace()
        open('email_status_queue/{}.json'.format(msg['MessageId']), 'w+').write(body)
        message.delete()
        sys.stdout.write('.')
        sys.stdout.flush()
        nprocessed = 1
    if nprocessed == 0:
        nempty += 1

[chadwhitacre]

127 messages.

[chadwhitacre]

Four bounces.

[chadwhitacre]

(But that's not what we're looking for here.)

[chadwhitacre]

Counts:

2017-02-10 2
2017-02-11 6
2017-02-12 9
2017-02-13 1
2017-02-14 3
2017-02-16 47
2017-02-17 9
2017-02-18 8
2017-02-19 1
2017-02-20 6
2017-02-21 5
2017-02-22 8
2017-02-23 7
2017-02-24 15

[chadwhitacre]

All of todays are connect notifications. There are no payday notifications.

[chadwhitacre]

Alright. So do we have SES API call failures in Papertrail?

[chadwhitacre]

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

import json
import os
from collections import defaultdict

base = 'email_status_queue'


by_date = defaultdict(list)


for filename in os.listdir(base):
    if not filename.endswith('.json'):
        continue
    msg = json.load(open(os.path.join(base, filename)))
    msg = json.loads(msg['Message'])
    mail = msg['mail']

    if msg['notificationType'] != 'Delivery':
        print(mail['timestamp'], msg['notificationType'], mail['destination'])

    date = mail['timestamp'][:10]
    by_date[date].append(msg)

print()
print('-'*78)
print()

groups = list(sorted(by_date.items()))

for date, messages in groups:
    print(date, len(messages))

for date, messages in sorted(by_date.items()):
    print()
    print(date, '-'*67)
    print()
    for msg in messages:
        subject = msg['mail']['commonHeaders']['subject']
        print(date, subject.encode('utf8'))

[chadwhitacre]

Oh! Also, payday was yesterday (2017-02-23), not today.

[chadwhitacre]

Did we deploy ... gratipay/gratipay.com#4170? No.

[chadwhitacre]

Nothing for amazon in Papertrail (Is that the right search term? Truncation?).

[chadwhitacre]

Let's check payday logs ...

[chadwhitacre]

pid-21004 thread-140079838471936 (MainThread) Notifying participants.

https://github.com/gratipay/logs/blob/master/payday/gratipay-247.log#L120

[chadwhitacre]

Looks like that's the expected output.

[chadwhitacre]

Here's the code.

[chadwhitacre]

Are the messages still in the queue?

[chadwhitacre]

Yes!

[chadwhitacre]

Sure would be nice to have a timestamp in that table. :-/

[chadwhitacre]

Alright, so immediate fix is supposedly to flush the queue.

What's the root cause?

[chadwhitacre]

I guess that means the dequeue thread isn't running? Or is missing these?

[chadwhitacre]

https://github.com/gratipay/gratipay.com/blob/2050/gratipay/main.py#L88

https://github.com/gratipay/gratipay.com/blob/2050/gratipay/models/participant/__init__.py#L575-L595

[chadwhitacre]

Was there an exception on the thread?

[chadwhitacre]

Would that make it into Sentry?

[chadwhitacre]

Yes! And, yes!

[chadwhitacre]

A bug in the Spanish translation, it looks like?

[chadwhitacre]

[chadwhitacre]

I'm having trouble finding the singular string. I've reconfigured permissions in Transifex so I can access the translation view from my personal account (vs. Gratipay account).

[chadwhitacre]

Got it!

[chadwhitacre]

Blam! Queue is empty. 👍

[chadwhitacre]

Ping @sseerrggii. Turns out that a bug lurking for the past two years in the Spanish translation of Gratipay caused billing notifications to fail today (details above on this ticket). I've updated "{0} y {1}" to "{0} y otro." If that's not correct please update in Transifex and let me know so I can redeploy. Thanks! :-)

[chadwhitacre]

Announced.

[chadwhitacre]

Four bounces.

#972 (comment) ...

[chadwhitacre]

Alright, they were all duds (either removed or never verified or account closed). Okay! 💃

[sseerrggii]

Ok @whit537 It seems fine to me :)