chore: email treasury report zip file #399

vshia · 2024-08-24T02:26:11Z

Need help on the Terraform setup for AWS SES

Task: #321

vshia · 2024-08-24T02:26:57Z

python/pyproject.toml

@@ -12,6 +12,8 @@ pydantic = "^2.6.4"
 aws-lambda-typing = "^2.19.0"
 boto3 = "^1.34.70"
 structlog = "^24.1.0"
+html2text = "^2024.2.26"


will remove

github-actions · 2024-08-24T02:38:36Z

QA Summary

See our documentation for tips on how to resolve failing QA checks.

QA Check	Result
🌐 Web Tests	✅
🔗 API Tests	✅
🐍 Python Tests	✅
📏 ESLint	✅
🧼 Ruff	✅
🧹 TFLint	✅

Test Coverage

Coverage report for api suite

St	File	% Stmts	% Branch	% Funcs	% Lines	Uncovered Line #s
🔴	All files	49.12	32.5	55.88	49.55
🔴	src	0	100	0	0
🔴	server.ts	0	100	0	0	6-13
🟢	src/directives/requireAuth	100	100	100	100
🟢	requireAuth.ts	100	100	100	100
🟡	src/directives/skipAuth	50	100	0	50
🟡	skipAuth.ts	50	100	0	50	13
🔴	src/functions	0	100	0	0
🔴	graphql.ts	0	100	0	0	15-27
🔴	src/functions/processValidationJson	38.55	50	44.44	37.8
🟢	processValidationJson.scenarios.ts	100	100	100	100
🔴	processValidationJson.ts	37.8	50	44.44	37.03	59-98,118-119,157-165,177-178,193-196,201-242,255-256,270-309,321-324,332
🔴	src/graphql	0	100	100	0
🔴	agencies.sdl.ts	0	100	100	0	1
🔴	expenditureCategories.sdl.ts	0	100	100	0	1
🔴	inputTemplates.sdl.ts	0	100	100	0	1
🔴	organizations.sdl.ts	0	100	100	0	1
🔴	outputTemplates.sdl.ts	0	100	100	0	1
🔴	projects.sdl.ts	0	100	100	0	1
🔴	reportingPeriodCertifications.sdl.ts	0	100	100	0	1
🔴	reportingPeriods.sdl.ts	0	100	100	0	1
🔴	subrecipientUploads.sdl.ts	0	100	100	0	1
🔴	subrecipients.sdl.ts	0	100	100	0	1
🔴	uploadValidations.sdl.ts	0	100	100	0	1
🔴	uploads.sdl.ts	0	100	100	0	1
🔴	users.sdl.ts	0	100	100	0	1
🔴	validationRuleses.sdl.ts	0	100	100	0	1
🔴	src/lib	13.12	9.57	11.7	13.43
🟡	auth.ts	62.96	48.48	57.14	65.38	60-61,77-78,84-85,101-102,124,131,134,139-146,170,174
🔴	aws.ts	25.42	18.75	25	25.42	53-58,74-97,121-123,150-171,186-272
🟢	constants.ts	100	100	100	100
🔴	db.ts	45.45	50	50	45.45	15-35,41,43,50
🔴	ec-codes.ts	0	100	100	0	1
🟢	logger.ts	100	100	100	100
🔴	persist-upload.js	0	0	0	0	16-295
🔴	preconditions.ts	0	0	0	0	2-3
🔴	records.js	0	0	0	0	12-214
🔴	templateRules.ts	0	0	0	0
🔴	tracer.ts	0	100	100	0	5-14
🔴	validate-upload.js	0	0	0	0	18-790
🟢	validation-error.ts	83.33	100	50	83.33	22
🔴	validation-rules.js	0	0	0	0	6-194
🟡	src/services/agencies	67.34	50	80	67.34
🟢	agencies.scenarios.ts	100	100	100	100
🟡	agencies.ts	65.21	50	75	65.21	40-51,60-64,97-98,104,113-121
🟡	src/services/expenditureCategories	78.57	66.66	88.88	78.57
🟢	expenditureCategories.scenarios.ts	100	100	100	100
🟡	expenditureCategories.ts	77.77	66.66	88.88	77.77	30-34,49-52,60,91
🟡	src/services/inputTemplates	77.77	66.66	85.71	77.77
🟢	inputTemplates.scenarios.ts	100	100	100	100
🟡	inputTemplates.ts	76.92	66.66	85.71	76.92	25-29,39-40,50,85
🟡	src/services/organizations	75.71	88.88	52.63	75.71
🟢	organizations.scenarios.ts	100	100	100	100
🟡	organizations.ts	74.62	88.88	47.05	74.62	36-40,75,147-177,185,203-230
🟢	src/services/outputTemplates	82.85	66.66	85.71	82.85
🟢	outputTemplates.scenarios.ts	100	100	100	100
🟢	outputTemplates.ts	82.35	66.66	85.71	82.35	26-30,40-41,51,114
🟡	src/services/passage	74.07	62.5	100	74.07
🟡	passage.ts	74.07	62.5	100	74.07	18-19,65-76
🟡	src/services/projects	80	100	62.5	80
🟢	projects.scenarios.ts	100	100	100	100
🟡	projects.ts	78.57	100	62.5	78.57	45-51
🟢	src/services/reportingPeriodCertifications	100	100	100	100
🟢	reportingPeriodCertifications.scenarios.ts	100	100	100	100
🟢	reportingPeriodCertifications.ts	100	100	100	100
🟡	src/services/reportingPeriods	67.5	70	50	67.5
🟢	reportingPeriods.scenarios.ts	100	100	100	100
🟡	reportingPeriods.ts	66.66	70	50	66.66	25-29,39-40,55-58,74,105-126
🟢	src/services/subrecipientUploads	88.88	83.33	85.71	88.88
🟢	subrecipientUploads.scenarios.ts	100	100	100	100
🟢	subrecipientUploads.ts	86.36	83.33	80	86.36	64,94-99
🟢	src/services/subrecipients	92.85	100	88.88	92.85
🟢	subrecipients.scenarios.ts	100	100	100	100
🟢	subrecipients.ts	90	100	81.81	90	53-58
🟡	src/services/uploadValidations	57.14	100	14.28	57.14
🟢	uploadValidations.scenarios.ts	100	100	100	100
🟡	uploadValidations.ts	53.84	100	14.28	53.84	10,16,30,38,45-48
🟢	src/services/uploads	91.34	71.42	88.63	91.34
🟢	uploads.scenarios.ts	100	100	100	100
🟢	uploads.ts	88.31	71.42	75	88.31	37,103,131-145,258-262
🟢	src/services/users	85.61	82	88.88	85.61
🟢	users.scenarios.ts	100	100	100	100
🟢	users.ts	84.61	82	84.21	84.61	222,239,255,277-279,288-292,310-311,325-328,346-348,356-357,362,371-377
🟢	src/services/validationRuleses	85.71	100	71.42	85.71
🟢	validationRuleses.scenarios.ts	100	100	100	100
🟢	validationRuleses.ts	84.61	100	71.42	84.61	43-48

Coverage report for web suite

St	File	% Stmts	% Branch	% Funcs	% Lines	Uncovered Line #s
🔴	All files	10.43	11.71	8.74	9.91
🟢	api/src/lib	100	100	100	100
🟢	constants.ts	100	100	100	100
🔴	web/src	28.57	18.75	66.66	28.57
🔴	App.tsx	0	0	0	0	3-36
🟢	Routes.tsx	100	100	100	100
🟡	auth.ts	50	50	100	50	19-24
🔴	entry.client.tsx	0	0	100	0	10-22
🔴	web/src/auth	7.14	0	4.16	7.14
🔴	localAuth.ts	9.09	0	8.33	9.09	39-68,76-80
🔴	passageAuth.ts	5	0	0	5	22-25,31-60
🔴	web/src/components/Agency/Agencies	0	100	0	0
🔴	Agencies.tsx	0	100	0	0	9-21
🔴	web/src/components/Agency/AgenciesCell	0	100	0	0
🔴	AgenciesCell.tsx	0	100	0	0	8-39
🔴	web/src/components/Agency/Agency	0	0	0	0
🔴	Agency.tsx	0	0	0	0	10-78
🔴	web/src/components/Agency/AgencyCell	0	100	0	0
🔴	AgencyCell.tsx	0	100	0	0	7-27
🔴	web/src/components/Agency/AgencyForm	0	0	0	0
🔴	AgencyForm.tsx	0	0	0	0	25-45
🔴	web/src/components/Agency/EditAgencyCell	0	100	0	0
🔴	EditAgencyCell.tsx	0	100	0	0	10-59
🔴	web/src/components/Agency/NewAgency	0	100	0	0
🔴	NewAgency.tsx	0	100	0	0	9-35
🟢	web/src/components/Navigation	100	60	100	100
🟢	Navigation.tsx	100	60	100	100	24-57
🔴	web/src/components/Organization/EditOrganizationCell	0	100	0	0
🔴	EditOrganizationCell.tsx	0	100	0	0	13-64
🔴	web/src/components/Organization/EditOrganizationForm	0	0	0	0
🔴	EditOrganizationForm.tsx	0	0	0	0	27-41
🔴	web/src/components/Organization/NewOrganization	0	100	0	0
🔴	NewOrganization.tsx	0	100	0	0	9-37
🔴	web/src/components/Organization/NewOrganizationForm	0	0	0	0
🔴	NewOrganizationForm.tsx	0	0	0	0	25-54
🔴	web/src/components/Organization/Organization	0	0	0	0
🔴	Organization.tsx	0	0	0	0	10-70
🔴	web/src/components/Organization/OrganizationCell	0	100	0	0
🔴	OrganizationCell.tsx	0	100	0	0	7-28
🔴	web/src/components/Organization/OrganizationPickListsCell	40	0	27.27	36.36
🟡	OrganizationPickListsCell.mock.ts	50	100	0	100
🔴	OrganizationPickListsCell.stories.tsx	0	0	0	0	6-32
🟡	OrganizationPickListsCell.tsx	64.28	100	50	58.33	14-16,47-70
🔴	web/src/components/Organization/Organizations	0	100	0	0
🔴	Organizations.tsx	0	100	0	0	9-21
🔴	web/src/components/Organization/OrganizationsCell	0	100	0	0
🔴	OrganizationsCell.tsx	0	100	0	0	8-37
🔴	web/src/components/OutputTemplate/EditOutputTemplateCell	0	100	0	0
🔴	EditOutputTemplateCell.tsx	0	100	0	0	18-81
🔴	web/src/components/OutputTemplate/NewOutputTemplate	0	0	0	0
🔴	NewOutputTemplate.tsx	0	0	0	0	17-126
🔴	web/src/components/OutputTemplate/OutputTemplate	0	0	0	0
🔴	OutputTemplate.tsx	0	0	0	0	17-97
🔴	web/src/components/OutputTemplate/OutputTemplateCell	0	100	0	0
🔴	OutputTemplateCell.tsx	0	100	0	0	17-47
🔴	web/src/components/OutputTemplate/OutputTemplateForm	0	0	0	0
🔴	OutputTemplateForm.tsx	0	0	0	0	18-63
🔴	web/src/components/OutputTemplate/OutputTemplates	0	0	0	0
🔴	OutputTemplates.tsx	0	0	0	0	18-94
🔴	web/src/components/OutputTemplate/OutputTemplatesCell	0	100	0	0
🔴	OutputTemplatesCell.tsx	0	100	0	0	18-52
🔴	web/src/components/ReportingPeriod/EditReportingPeriodCell	0	100	0	0
🔴	EditReportingPeriodCell.tsx	0	100	0	0	13-74
🔴	web/src/components/ReportingPeriod/NewReportingPeriod	0	100	0	0
🔴	NewReportingPeriod.tsx	0	100	0	0	9-35
🔴	web/src/components/ReportingPeriod/ReportingPeriod	0	0	0	0
🔴	ReportingPeriod.tsx	0	0	0	0	12-101
🔴	web/src/components/ReportingPeriod/ReportingPeriodCell	0	100	0	0
🔴	ReportingPeriodCell.tsx	0	100	0	0	7-33
🔴	web/src/components/ReportingPeriod/ReportingPeriodForm	0	0	0	0
🔴	ReportingPeriodForm.tsx	0	0	0	0	18-43
🔴	web/src/components/ReportingPeriod/ReportingPeriods	0	0	0	0
🔴	ReportingPeriods.tsx	0	0	0	0	13-96
🔴	web/src/components/ReportingPeriod/ReportingPeriodsCell	0	100	0	0
🔴	ReportingPeriodsCell.tsx	0	100	0	0	8-43
🟡	web/src/components/ReportingPeriodsCell	57.14	0	60	50
🟢	ReportingPeriodsCell.mock.ts	100	100	100	100
🔴	ReportingPeriodsCell.stories.tsx	0	0	0	0	6-32
🟢	ReportingPeriodsCell.tsx	100	100	100	100
🔴	web/src/components/Subrecipient/SubrecipientTableUploadLinksDisplay	0	0	0	0
🔴	SubrecipientTableUploadLinksDisplay.stories.tsx	0	100	100	0	5-82
🔴	SubrecipientTableUploadLinksDisplay.tsx	0	0	0	0	5-77
🔴	web/src/components/Subrecipient/Subrecipients	0	0	0	0
🔴	Subrecipients.tsx	0	100	0	0	5-8
🔴	columns.tsx	0	0	0	0	7-93
🔴	web/src/components/Subrecipient/SubrecipientsCell	0	100	0	0
🔴	SubrecipientsCell.tsx	0	100	0	0	7-62
🔴	web/src/components/TableBuilder	0	0	0	0
🔴	DebouncedInput.tsx	0	0	0	0	13-32
🔴	Filter.tsx	0	0	0	0	6-15
🔴	TableBuilder.tsx	0	0	0	0	22-70
🔴	TableHeader.tsx	0	0	0	0	5-42
🔴	TableRow.tsx	0	100	0	0	3-7
🟡	web/src/components/TemplateUploadReportingPeriodCell	55	0	55.55	47.05
🟢	TemplateUploadReportingPeriodCell.mock.ts	100	100	100	100
🔴	TemplateUploadReportingPeriodCell.stories.tsx	0	0	0	0	11-37
🟢	TemplateUploadReportingPeriodCell.tsx	100	100	100	100
🔴	web/src/components/TreasuryGeneration/DownloadTreasuryFiles	0	100	0	0
🔴	DownloadTreasuryFiles.tsx	0	100	0	0	6-30
🔴	web/src/components/TreasuryGeneration/NewTreasuryGeneration	0	100	0	0
🔴	NewTreasuryGeneration.tsx	0	100	0	0	8-39
🔴	web/src/components/TreasuryGeneration/NewTreasuryGenerationForm	0	0	0	0
🔴	NewTreasuryGenerationForm.tsx	0	0	0	0	20-31
🔴	web/src/components/Upload/EditUploadCell	0	100	0	0
🔴	EditUploadCell.tsx	0	100	0	0	10-66
🔴	web/src/components/Upload/NewUpload	0	100	0	0
🔴	NewUpload.tsx	...[Comment body truncated]

as1729

@vshia I have requested a few changes on the existing code. I will leave comments about Amazon SES after testing it out locally first by EOD wednesday.

as1729 · 2024-08-27T16:48:27Z

python/src/functions/generate_presigned_url_and_send_email.py

+    1) Check to see if the s3 object exists:
+        treasuryreports/{organization.id}/{organization.preferences.current_reporting_period_id}/report.zip
+    2) If it does not, raise an exception and quit
+    3) Generate a pre-signed URL (what's the expiration date?)


I think 1 hour should be sufficient for this.

python/src/lib/email.py

terraform/functions.tf

python/src/functions/generate_presigned_url_and_send_email.py

as1729

@vshia I added a comment indicating the removal of a few resources. Later today I will provide you with a code-snippet of what resources or in our case data that we need to add here.

terraform/email.tf

terraform/variables.tf

as1729 · 2024-08-29T15:23:08Z

@vshia this PR contains the changes I would make to this PR in order to get the SES piece to work: #405

* chore: reorganizes email-lambda and provides ses permission * chore: remove the extra configuration related to emails

github-actions · 2024-08-31T07:51:59Z

Terraform Summary

Step	Result
🖌 Terraform Format & Style	✅
⚙️ Terraform Initialization	✅
🤖 Terraform Validation	✅
📖 Terraform Plan	✅

Hint: If "Terraform Format & Style" failed, run terraform fmt -recursive from the terraform/ directory and commit the results.

Output

Validation Output

Success! The configuration is valid.

Plan Summary

CHANGE	RESOURCE
add	`aws_s3_object.origin_dist_artifact["static/js/app.0de605db.js"]`
	`aws_s3_object.origin_dist_artifact["static/js/app.0de605db.js.LICENSE.txt"]`
	`module.lambda_function-email-presigned-url.aws_cloudwatch_log_group.lambda[0]`
	`module.lambda_function-email-presigned-url.aws_iam_policy.additional_inline[0]`
	`module.lambda_function-email-presigned-url.aws_iam_policy.additional_jsons[0]`
	`module.lambda_function-email-presigned-url.aws_iam_policy.logs[0]`
	`module.lambda_function-email-presigned-url.aws_iam_role.lambda[0]`
	`module.lambda_function-email-presigned-url.aws_iam_role_policy_attachment.additional_inline[0]`
	`module.lambda_function-email-presigned-url.aws_iam_role_policy_attachment.additional_jsons[0]`
	`module.lambda_function-email-presigned-url.aws_iam_role_policy_attachment.logs[0]`
	`module.lambda_function-email-presigned-url.aws_lambda_function.this[0]`
	`module.lambda_function-email-presigned-url.aws_lambda_permission.current_version_triggers["StepFunctionTrigger"]`
	`module.lambda_function-email-presigned-url.aws_lambda_permission.unqualified_alias_triggers["StepFunctionTrigger"]`
update	`aws_ecs_service.console`
	`aws_s3_object.origin_dist_artifact["200.html"]`
	`aws_s3_object.origin_dist_artifact["build-manifest.json"]`
	`aws_s3_object.origin_dist_artifact["chunk-references.json"]`
	`aws_s3_object.origin_dist_artifact["index.html"]`
	`module.lambda_function-cpfCreateArchive.aws_lambda_function.this[0]`
	`module.lambda_function-cpfValidation.aws_lambda_function.this[0]`
	`module.lambda_function-graphql.aws_lambda_function.this[0]`
	`module.lambda_function-processValidationJson.aws_lambda_function.this[0]`
	`module.lambda_function-subrecipientTreasuryReportGen.aws_lambda_function.this[0]`
	`module.lambda_function-treasuryProjectFileGeneration.aws_lambda_function.this[0]`
recreate	`aws_ecs_task_definition.console`
	`aws_s3_object.lambda_artifact-graphql`
	`aws_s3_object.lambda_artifact-processValidationJson`
	`aws_s3_object.lambda_artifact-python`
	`module.lambda_function-cpfCreateArchive.aws_lambda_permission.current_version_triggers["StepFunctionTrigger"]`
	`module.lambda_function-cpfValidation.aws_lambda_permission.current_version_triggers["S3BucketNotification"]`
	`module.lambda_function-graphql.aws_lambda_permission.current_version_triggers["APIGateway"]`
	`module.lambda_function-processValidationJson.aws_lambda_permission.current_version_triggers["S3BucketNotification"]`
	`module.lambda_function-subrecipientTreasuryReportGen.aws_lambda_permission.current_version_triggers["StepFunctionTrigger"]`
	`module.lambda_function-treasuryProjectFileGeneration.aws_lambda_permission.current_version_triggers["S3BucketNotification"]`
	`module.lambda_function-treasuryProjectFileGeneration.aws_lambda_permission.current_version_triggers["StepFunctionTrigger"]`
delete	`aws_s3_object.origin_dist_artifact["static/js/app.9c7c448e.js"]`
	`aws_s3_object.origin_dist_artifact["static/js/app.9c7c448e.js.LICENSE.txt"]`

Pusher: @vshia, Action: pull_request_target, Workflow: Continuous Integration

as1729

Some additional changes @vshia. Mainly related to typing, naming, and error handling. Ideally if we can have custom error message types, that would be the best so we are can more easily identify and triage errors when we see them in datadog, but for now having descriptive error messages will do.

python/src/functions/generate_presigned_url_and_send_email.py

python/src/lib/s3_helper.py

python/src/functions/generate_presigned_url_and_send_email.py

…-reporter into rev/treasury_email

as1729 · 2024-09-16T11:25:44Z

Thanks @vshia looks great!

chore: email treasury report

cd1255e

vshia self-assigned this Aug 24, 2024

vshia commented Aug 24, 2024

View reviewed changes

github-actions bot added infra python terraform labels Aug 24, 2024

as1729 requested changes Aug 27, 2024

View reviewed changes

as1729 reviewed Aug 28, 2024

View reviewed changes

terraform/email.tf Outdated Show resolved Hide resolved

terraform/email.tf Outdated Show resolved Hide resolved

terraform/email.tf Outdated Show resolved Hide resolved

terraform/email.tf Outdated Show resolved Hide resolved

terraform/variables.tf Outdated Show resolved Hide resolved

as1729 and others added 5 commits August 30, 2024 17:58

chore: reorganizes email-lambda and provides ses permission (#405)

e0f1a1b

* chore: reorganizes email-lambda and provides ses permission * chore: remove the extra configuration related to emails

Minor changes

d7a16be

Update poetry.lock

72dcb0e

Get email html to work

20dffca

Merge remote-tracking branch 'origin/main' into rev/treasury_email

02d0121

github-actions bot added the dependencies label Aug 31, 2024

vshia added 2 commits September 3, 2024 21:23

fix lint

fc63a5e

chore: Remove duplicate import

62bde3d

vshia requested a review from as1729 September 4, 2024 05:07

vshia marked this pull request as ready for review September 4, 2024 05:07

vshia added 3 commits September 5, 2024 11:53

Merge branch 'main' into rev/treasury_email

f6129b2

Merge branch 'main' into rev/treasury_email

b56eb3f

Merge branch 'main' into rev/treasury_email

e94a087

as1729 reviewed Sep 12, 2024

View reviewed changes

vshia added 2 commits September 13, 2024 09:37

Update typing, naming and error handling

0d387fd

Merge branch 'rev/treasury_email' of github.com:usdigitalresponse/cpf…

9036fc8

…-reporter into rev/treasury_email

vshia requested a review from as1729 September 13, 2024 18:06

as1729 approved these changes Sep 16, 2024

View reviewed changes

as1729 merged commit ed30f13 into main Sep 16, 2024
22 checks passed

as1729 deleted the rev/treasury_email branch September 16, 2024 11:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: email treasury report zip file #399

chore: email treasury report zip file #399

vshia commented Aug 24, 2024 •

edited

Loading

vshia Aug 24, 2024

github-actions bot commented Aug 24, 2024 •

edited

Loading

as1729 left a comment

as1729 Aug 27, 2024

as1729 left a comment

as1729 commented Aug 29, 2024

github-actions bot commented Aug 31, 2024 •

edited

Loading

as1729 left a comment

as1729 commented Sep 16, 2024

chore: email treasury report zip file #399

chore: email treasury report zip file #399

Conversation

vshia commented Aug 24, 2024 • edited Loading

vshia Aug 24, 2024

Choose a reason for hiding this comment

github-actions bot commented Aug 24, 2024 • edited Loading

QA Summary

Test Coverage

as1729 left a comment

Choose a reason for hiding this comment

as1729 Aug 27, 2024

Choose a reason for hiding this comment

as1729 left a comment

Choose a reason for hiding this comment

as1729 commented Aug 29, 2024

github-actions bot commented Aug 31, 2024 • edited Loading

Terraform Summary

Output

as1729 left a comment

Choose a reason for hiding this comment

as1729 commented Sep 16, 2024

vshia commented Aug 24, 2024 •

edited

Loading

github-actions bot commented Aug 24, 2024 •

edited

Loading

github-actions bot commented Aug 31, 2024 •

edited

Loading