QA - Performance testing C&P integration

[cris-oddball]

User Story - Business Need

Based on an incident during the November C&P run, timings were changed in celery for processing messages out of the DynamoDB table and logging statements were added to be able to know how long certain DynamoDB connections took.

• Ticket is understood, and QA has been contacted (if the ticket has a QA label).

User Story(ies)

As a VANotify engineer
I want to be able to track the DynamoDB connection timings
So that I can make appropriate changes based on timings

As a VANotify engineer
I want to be know how long it will take to process ~70k notifications
So that I can make appropriate changes based on timings

Additional Info and Resources

• Deploy this API PR to Perf (always keep it up to date with main)
• Deploy this Infra branch to Perf (always keep it up to date with main) for it to match the API/celery tasks and auto-scaling on Prod
• Note that we did a short test on perf first to be able to create facets, but were unable to create the necessary facets as we were not sure how to configure them.
  
  
• Note that we also set the Feature Flag to disabled in Perf because we send everything to one phone number (the AWS simulated number and receive 429s from Pinpoint because it balks at sending mass notifications to a single destination. I had created an AWS simulated originator number, which worked in the test console (guarantees delivery without actually sending anything), however it failed to send anything via Postman, which factored into the decision to disable the feature to send notifications during the test run.

deliver_sms: Notification: a305140a-6603-49c8-81a1-b9cce6b4bfe3 - Experienced an exception: StatusCode: 400, StatusMessage:The provided number does not exist or does not belong to the account

{"event_type":"_SMS.FAILURE","event_timestamp":1734446590450,"arrival_timestamp":1734446590450,"event_version":"3.1","application":{"app_id":"<redacted>","sdk":{}},"client":{"client_id":"<redacted>"},"device":{"platform":{}},"session":{},"attributes":{"sender_request_id":"37ac0914-d88b-4b73-bb46-0e4a53e115e0","destination_phone_number":"+14254147755","record_status":"INVALID_PHONE_NUMBER","iso_country_code":"US","number_of_message_parts":"1","message_id":"5sgekdnesscmbabdggn9ua4077lt51kuesgrmi80","message_type":"Transactional"},"metrics":{"price_in_millicents_usd":0.0},"awsAccountId":"<redacted>"}

Acceptance Criteria

• A performance test with 70k messages is run in Perf
• This work is added to the sprint review slide deck (key win bullet point and demo slide)

QA Considerations

See comments for results.

Potential Dependencies

[cris-oddball]

RESULTS

• kicked off at 12:33 PM MT
• finished at 1:29 AM MT
• 69,915 in DynamoDB table - all were processed
• 69.9k logs for "updated record from dynamodb"
• no errors seen related to C&P during that time.

Despite not having the facets created, there is some information that can be gleaned from Datadog (screenshots to follow)

[cris-oddball]

[cris-oddball]

Need to update Wiki with how to run the performance test now that it's been revised, will then close this ticket.

[cris-oddball]

The wiki has been updated for running the Performance test.