Upgrade our stack to latest Python 3.12

[jimleroyer]

Description

As a software developer working on GCNotify stack,
I want the Python stack to be upgraded at latest supported version,
So that I can leverage the latest features of the language and better/safer code.

As a system ops working on GCNotify stack,
I want the Python stack to be upgraded at latest supported version,
So that I can leverage the latest performance improvement
And process notifications faster.

WHY are we building?

To get latest performance improvements, features and security updates of the Python language/runtime.

Several improvements from Python 3.10 to 3.12 will benefit us:

• Speed improvements
• Language improvements
• Security and bug fixes
• Library upgrades

WHAT are we building?

Upgrade the stack to the latest possible version of Python. A few considerations:

• It is possible that we cannot upgrade to the latest of the latest (3.12 at the time of this writing for the stable version), as this might be blocked by a dependency library that did not release a version supporting the latest. For example, we might have to upgrade from version 3.10 to version 3.11, and wait a bit longer to migrate to version 3.12.
• The lambdas might have additional considerations that might limit the possible maximum Python version we can use, such as supported runtime and also monitoring libraries that need to support that environment (such as New Relic). Consider doing this work in Upgrade lambda and New Relic to latest supported version #354 .

VALUE created by our solution

Better performance, security and reliability.

Acceptance Criteria

• utils 2
• document-download-api 2
• admin 2
• api 3
• The Kubernetes components running on Python are upgraded to Python version 3.12.

QA Steps

• Run load tests and compare performance to 3.10 (about 5500 emails / min max sustained)
• Smoke tests are ran against newly upgraded lambdas.
• Performance tests are ran against newly upgraded lambdas.
• Run rollercoaster tests
• Run a bug bash with the team to test all upgraded components.

Run in staging for at least a few days to ensure it's sane before pushing to production

[sastels]

Looking at moving document-download-api from 3.10 to 3.12
https://app.zenhub.com/workspaces/notify-planning-614b3ad91bc2030015ed22f5/issues/gh/cds-snc/notification-document-download-api/195

need utils before document-download-api of course :/
https://app.zenhub.com/workspaces/notify-planning-614b3ad91bc2030015ed22f5/issues/gh/cds-snc/notification-utils/326

[sastels]

also admin WIP cds-snc/notification-admin#1996

[sastels]

also started on api

[sastels]

fixing many broken tests 😭

[sastels]

Tests passing, ran admin / api / dd-api locally and all working! Going to give the changes another read through and then put for review.

[sastels]

Will convert this card to an epic and create separate cards for the 4 repos.
utils 2
document-download-api 2
admin 2
api 3

[sastels]

Documented how to build and run admin and dd-api on dev. dd-api crashed on start :/ This is a problem for Thursday Steve.

Thursday Steve should also break up this card.

[jimleroyer]

It seems the DD-API pod restart once in a while even without the Python upgrade. Steve will double-validate and create a card to follow up on that if that's the case.

[sastels]

(main branch) dd-api pods not restarting on prod, this might just be a dev issue.