Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add ability to delete and reset testfiles #346

Merged
merged 3 commits into from
Jun 28, 2024
Merged

feat: add ability to delete and reset testfiles #346

merged 3 commits into from
Jun 28, 2024

Conversation

as1729
Copy link
Contributor

@as1729 as1729 commented Jun 28, 2024 •
edited
Loading

#333

This PR ensures that we can delete the test upload files that partners have added in production. Running the script does the following:

  1. Identifies all uploads for the organization (optionally filtered by IDs if passed in)
  2. Deletes the UploadValidation record
  3. Deletes the Upload record
  4. Deletes the file from s3

Unit tests have been added and ran the script locally.

@github-actions github-actions bot added enhancement New feature or request javascript labels Jun 28, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 28, 2024 •
edited
Loading

QA Summary

See our documentation for tips on how to resolve failing QA checks.

QA Check Result
🌐 Web Tests ✅
🔗 API Tests ✅
🐍 Python Tests ✅
📏 ESLint ✅
🧼 Ruff ✅
🧹 TFLint ✅

Test Coverage

Coverage report for api suite
St File % Stmts % Branch % Funcs % Lines Uncovered Line #s
🔴 All files 43.37 25.25 47.98 43.81
🔴  src 0 100 0 0
🔴   server.ts 0 100 0 0 6-13
🟢  src/directives/requireAuth 100 100 100 100
🟢   requireAuth.ts 100 100 100 100
🟡  src/directives/skipAuth 50 100 0 50
🟡   skipAuth.ts 50 100 0 50 13
🔴  src/functions 0 100 0 0
🔴   graphql.ts 0 100 0 0 15-27
🟡  src/functions/processValidationJson 56.86 50 60 56
🟢   processValidationJson.scenarios.ts 100 100 100 100
🟡   processValidationJson.ts 56 50 60 55.1 48-87,107-108,144-152,164-165,180-183,195-196,209
🔴  src/graphql 0 100 100 0
🔴   agencies.sdl.ts 0 100 100 0 1
🔴   expenditureCategories.sdl.ts 0 100 100 0 1
🔴   inputTemplates.sdl.ts 0 100 100 0 1
🔴   organizations.sdl.ts 0 100 100 0 1
🔴   outputTemplates.sdl.ts 0 100 100 0 1
🔴   projects.sdl.ts 0 100 100 0 1
🔴   reportingPeriods.sdl.ts 0 100 100 0 1
🔴   subrecipientUploads.sdl.ts 0 100 100 0 1
🔴   subrecipients.sdl.ts 0 100 100 0 1
🔴   uploadValidations.sdl.ts 0 100 100 0 1
🔴   uploads.sdl.ts 0 100 100 0 1
🔴   users.sdl.ts 0 100 100 0 1
🔴   validationRuleses.sdl.ts 0 100 100 0 1
🔴  src/lib 10.37 8.27 7.77 10.63
🟡   auth.ts 64.7 48.38 57.14 67.34 70-71,77-78,94-95,117,124,127,132-139,163,167
🔴   aws.ts 2.17 0 0 2.17 29-223
🟢   constants.ts 100 100 100 100
🔴   db.ts 45.45 50 50 45.45 15-35,41,43,50
🔴   ec-codes.ts 0 100 100 0 1
🟢   logger.ts 100 100 100 100
🔴   persist-upload.js 0 0 0 0 16-295
🔴   preconditions.ts 0 0 0 0 2-3
🔴   records.js 0 0 0 0 12-214
🔴   templateRules.ts 0 0 0 0
🔴   tracer.ts 0 100 100 0 5-14
🔴   validate-upload.js 0 0 0 0 18-790
🟢   validation-error.ts 83.33 100 50 83.33 22
🔴   validation-rules.js 0 0 0 0 6-194
🟡  src/services/agencies 67.34 50 80 67.34
🟢   agencies.scenarios.ts 100 100 100 100
🟡   agencies.ts 65.21 50 75 65.21 40-51,60-64,97-98,104,113-121
🟡  src/services/expenditureCategories 78.57 66.66 88.88 78.57
🟢   expenditureCategories.scenarios.ts 100 100 100 100
🟡   expenditureCategories.ts 77.77 66.66 88.88 77.77 30-34,49-52,60,91
🟡  src/services/inputTemplates 77.77 66.66 85.71 77.77
🟢   inputTemplates.scenarios.ts 100 100 100 100
🟡   inputTemplates.ts 76.92 66.66 85.71 76.92 25-29,39-40,50,85
🟡  src/services/organizations 67.39 87.5 50 67.39
🟢   organizations.scenarios.ts 100 100 100 100
🟡   organizations.ts 65.11 87.5 42.85 65.11 34-38,73,86-116,139-163
🟡  src/services/outputTemplates 77.77 66.66 85.71 77.77
🟢   outputTemplates.scenarios.ts 100 100 100 100
🟡   outputTemplates.ts 76.92 66.66 85.71 76.92 25-29,39-40,50,81
🟡  src/services/passage 74.07 62.5 100 74.07
🟡   passage.ts 74.07 62.5 100 74.07 18-19,65-76
🟡  src/services/projects 80 100 62.5 80
🟢   projects.scenarios.ts 100 100 100 100
🟡   projects.ts 78.57 100 62.5 78.57 45-51
🟡  src/services/reportingPeriods 67.5 70 50 67.5
🟢   reportingPeriods.scenarios.ts 100 100 100 100
🟡   reportingPeriods.ts 66.66 70 50 66.66 25-29,39-40,55-58,74,105-126
🟢  src/services/subrecipientUploads 92.3 100 83.33 92.3
🟢   subrecipientUploads.scenarios.ts 100 100 100 100
🟢   subrecipientUploads.ts 91.66 100 83.33 91.66 46
🟢  src/services/subrecipients 85.71 100 71.42 85.71
🟢   subrecipients.scenarios.ts 100 100 100 100
🟢   subrecipients.ts 84.61 100 71.42 84.61 47-52
🟡  src/services/uploadValidations 57.14 100 14.28 57.14
🟢   uploadValidations.scenarios.ts 100 100 100 100
🟡   uploadValidations.ts 53.84 100 14.28 53.84 10,16,30,38,45-48
🟢  src/services/uploads 86 66.66 78.26 86
🟢   uploads.scenarios.ts 100 100 100 100
🟢   uploads.ts 81.57 66.66 58.33 81.57 30,96,122-136
🟢  src/services/users 85.96 79.41 92 85.96
🟢   users.scenarios.ts 100 100 100 100
🟢   users.ts 84.9 79.41 88.88 84.9 227-229,238-242,260-261,275-278,296-298,306-307,312,321-324
🟢  src/services/validationRuleses 85.71 100 71.42 85.71
🟢   validationRuleses.scenarios.ts 100 100 100 100
🟢   validationRuleses.ts 84.61 100 71.42 84.61 43-48
Coverage report for web suite
St File % Stmts % Branch % Funcs % Lines Uncovered Line #s
🔴 All files 13.82 14.43 11.52 13.17
🟢  api/src/lib 100 100 100 100
🟢   constants.ts 100 100 100 100
🔴  web/src 28.57 18.75 66.66 28.57
🔴   App.tsx 0 0 0 0 3-36
🟢   Routes.tsx 100 100 100 100
🟡   auth.ts 50 50 100 50 19-24
🔴   entry.client.tsx 0 0 100 0 10-22
🔴  web/src/auth 7.14 0 4.16 7.14
🔴   localAuth.ts 9.09 0 8.33 9.09 39-68,76-80
🔴   passageAuth.ts 5 0 0 5 22-25,31-60
🔴  web/src/components/Agency/Agencies 0 100 0 0
🔴   Agencies.tsx 0 100 0 0 9-21
🔴  web/src/components/Agency/AgenciesCell 0 100 0 0
🔴   AgenciesCell.tsx 0 100 0 0 8-39
🔴  web/src/components/Agency/Agency 0 0 0 0
🔴   Agency.tsx 0 0 0 0 10-78
🔴  web/src/components/Agency/AgencyCell 0 100 0 0
🔴   AgencyCell.tsx 0 100 0 0 7-27
🔴  web/src/components/Agency/AgencyForm 0 0 0 0
🔴   AgencyForm.tsx 0 0 0 0 25-45
🔴  web/src/components/Agency/EditAgencyCell 0 100 0 0
🔴   EditAgencyCell.tsx 0 100 0 0 10-59
🔴  web/src/components/Agency/NewAgency 0 100 0 0
🔴   NewAgency.tsx 0 100 0 0 9-35
🟢  web/src/components/Navigation 100 37.5 100 100
🟢   Navigation.tsx 100 37.5 100 100 26-70
🔴  web/src/components/Organization/EditOrganizationCell 0 100 0 0
🔴   EditOrganizationCell.tsx 0 100 0 0 13-62
🔴  web/src/components/Organization/EditOrganizationForm 0 0 0 0
🔴   EditOrganizationForm.tsx 0 0 0 0 27-41
🔴  web/src/components/Organization/NewOrganization 0 100 0 0
🔴   NewOrganization.tsx 0 100 0 0 9-37
🔴  web/src/components/Organization/NewOrganizationForm 0 0 0 0
🔴   NewOrganizationForm.tsx 0 0 0 0 25-54
🔴  web/src/components/Organization/Organization 0 0 0 0
🔴   Organization.tsx 0 0 0 0 10-70
🔴  web/src/components/Organization/OrganizationCell 0 100 0 0
🔴   OrganizationCell.tsx 0 100 0 0 7-27
🔴  web/src/components/Organization/OrganizationPickListsCell 40 0 27.27 36.36
🟡   OrganizationPickListsCell.mock.ts 50 100 0 100
🔴   OrganizationPickListsCell.stories.tsx 0 0 0 0 6-32
🟡   OrganizationPickListsCell.tsx 64.28 100 50 58.33 14-16,51-76
🔴  web/src/components/Organization/Organizations 0 100 0 0
🔴   Organizations.tsx 0 100 0 0 9-21
🔴  web/src/components/Organization/OrganizationsCell 0 100 0 0
🔴   OrganizationsCell.tsx 0 100 0 0 8-37
🔴  web/src/components/ReportingPeriod/EditReportingPeriodCell 0 100 0 0
🔴   EditReportingPeriodCell.tsx 0 100 0 0 13-76
🔴  web/src/components/ReportingPeriod/NewReportingPeriod 0 100 0 0
🔴   NewReportingPeriod.tsx 0 100 0 0 9-35
🔴  web/src/components/ReportingPeriod/ReportingPeriod 0 0 0 0
🔴   ReportingPeriod.tsx 0 0 0 0 12-105
🔴  web/src/components/ReportingPeriod/ReportingPeriodCell 0 100 0 0
🔴   ReportingPeriodCell.tsx 0 100 0 0 7-34
🔴  web/src/components/ReportingPeriod/ReportingPeriodForm 0 0 0 0
🔴   ReportingPeriodForm.tsx 0 0 0 0 18-43
🔴  web/src/components/ReportingPeriod/ReportingPeriods 0 0 0 0
🔴   ReportingPeriods.tsx 0 0 0 0 13-98
🔴  web/src/components/ReportingPeriod/ReportingPeriodsCell 0 100 0 0
🔴   ReportingPeriodsCell.tsx 0 100 0 0 8-44
🟡  web/src/components/ReportingPeriodsCell 57.14 0 60 50
🟢   ReportingPeriodsCell.mock.ts 100 100 100 100
🔴   ReportingPeriodsCell.stories.tsx 0 0 0 0 6-32
🟢   ReportingPeriodsCell.tsx 100 100 100 100
🔴  web/src/components/TableBuilder 0 0 0 0
🔴   DebouncedInput.tsx 0 0 0 0 13-32
🔴   Filter.tsx 0 0 0 0 6-15
🔴   TableBuilder.tsx 0 0 0 0 22-70
🔴   TableHeader.tsx 0 0 0 0 5-42
🔴   TableRow.tsx 0 100 0 0 3-7
🟡  web/src/components/TemplateUploadReportingPeriodCell 55 0 55.55 47.05
🟢   TemplateUploadReportingPeriodCell.mock.ts 100 100 100 100
🔴   TemplateUploadReportingPeriodCell.stories.tsx 0 0 0 0 11-37
🟢   TemplateUploadReportingPeriodCell.tsx 100 100 100 100
🔴  web/src/components/Upload/EditUploadCell 0 100 0 0
🔴   EditUploadCell.tsx 0 100 0 0 10-66
🔴  web/src/components/Upload/NewUpload 0 100 0 0
🔴   NewUpload.tsx 0 100 0 0 7-35
🔴  web/src/components/Upload/Upload 0 0 0 0
🔴   Upload.stories.tsx 0 100 100 0 5-93
🔴   Upload.tsx 0 0 0 0 11-61
🔴  web/src/components/Upload/UploadCell 0 100 0 0
🔴   UploadCell.tsx 0 100 0 0 7-59
🔴  web/src/components/Upload/UploadForm 0 0 0 0
🔴   UploadForm.tsx 0 0 0 0 23-101
🔴  web/src/components/Upload/UploadValidationButtonGroup 0 0 0 0
🔴   UploadValidationButtonGroup.stories.tsx 0 100 0 0 5-47
🔴   UploadValidationButtonGroup.tsx 0 0 0 0 24-57
🔴  web/src/components/Upload/UploadValidationResultsTable 42.85 100 100 42.85
🔴   UploadValidationResultsTable.stories.tsx 0 100 100 0 6-41
🟢   UploadValidationResultsTable.tsx 100 100 100 100
🔴  web/src/components/Upload/UploadValidationStatus 0 0 0 0
🔴   UploadValidationStatus.tsx 0 0 0 0 3-30
🔴  web/src/components/Upload/Uploads 0 0 0 0
🔴   Uploads.tsx 0 100 0 0 9-17
🔴   columns.tsx 0 0 0 0 7-67
🔴  web/src/components/Upload/UploadsCell 0 100 0 0
🔴   UploadsCell.tsx 0 100 0 0 8-55
🔴  web/src/components/User/EditUserCell 0 100 0 0
🔴   EditUserCell.tsx 0 100 0 0 10-58
🔴  web/src/components/User/NewUser 0 100 0 0
🔴   NewUser.tsx 0 100 0 0 9-32
🔴  web/src/components/User/User 0 0 0 0
🔴   User.tsx 0 0 0 0 10-94
🔴  web/src/components/User/UserCell 0 100 0 0
🔴   UserCell.tsx 0 100 0 0 7-30
🔴  web/src/components/User/UserForm 0 0 0 0
🔴   UserForm.tsx 0 0 0 0 28-202
🔴  web/src/components/User/Users 0 100 0 0
🔴   Users.tsx 0 100 0 0 9-24
🔴  web/src/components/User/UsersCell 0 100 0 0
🔴   UsersCell.tsx 0 100 0 0 8-44
🟡  web/src/layouts/AuthenticatedLayout 60 50 100 60
🔴   AuthenticatedLayout.stories.tsx 0 100 100 0 5-13
🟢   AuthenticatedLayout.tsx 100 50 100 100 24
🟢  web/src/lib 100 100 100 100
🟢   formatters.tsx 100 100 100 100
🟢   seeds.ts 100 100 100 100
🔴  web/src/pages/Agency/AgenciesPage 0 100 0 0
🔴   AgenciesPage.tsx 0 100 0 0 8-12
🔴  web/src/pages/Agency/AgencyPage 0 100 0 0
🔴   AgencyPage.tsx 0 100 0 0 7-8
🔴  web/src/pages/Agency/EditAgencyPage 0 100 0 0
🔴   EditAgencyPage.tsx 0 100 0 0 7-8
🔴  web/src/pages...[Comment body truncated]

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 28, 2024 •
edited
Loading

Terraform Summary

Step Result
🖌 Terraform Format & Style ✅
⚙️ Terraform Initialization ✅
🤖 Terraform Validation ✅
📖 Terraform Plan ✅

Hint: If "Terraform Format & Style" failed, run terraform fmt -recursive from the terraform/ directory and commit the results.

Output

Validation Output 
Success! The configuration is valid.
Plan Summary
CHANGE RESOURCE
add aws_s3_object.origin_dist_artifact["static/js/app.d1f15f6d.js"]
aws_s3_object.origin_dist_artifact["static/js/app.d1f15f6d.js.LICENSE.txt"]
update aws_ecs_service.console
aws_s3_object.origin_dist_artifact["200.html"]
aws_s3_object.origin_dist_artifact["build-manifest.json"]
aws_s3_object.origin_dist_artifact["chunk-references.json"]
aws_s3_object.origin_dist_artifact["index.html"]
module.lambda_function-cpfValidation.aws_lambda_function.this[0]
module.lambda_function-graphql.aws_lambda_function.this[0]
module.lambda_function-processValidationJson.aws_lambda_function.this[0]
recreate aws_ecs_task_definition.console
aws_s3_object.lambda_artifact-graphql
aws_s3_object.lambda_artifact-processValidationJson
aws_s3_object.lambda_artifact-python
module.lambda_function-cpfValidation.aws_lambda_permission.current_version_triggers["S3BucketNotification"]
module.lambda_function-graphql.aws_lambda_permission.current_version_triggers["APIGateway"]
module.lambda_function-processValidationJson.aws_lambda_permission.current_version_triggers["S3BucketNotification"]
delete aws_s3_object.origin_dist_artifact["static/js/app.0b2fa856.js"]
aws_s3_object.origin_dist_artifact["static/js/app.0b2fa856.js.LICENSE.txt"]

Pusher: @as1729, Action: pull_request_target, Workflow: Continuous Integration

@as1729 as1729 enabled auto-merge (squash) June 28, 2024 20:29
laurelkorwin
laurelkorwin reviewed Jun 28, 2024
View reviewed changes
api/src/lib/aws.ts
@@ -114,6 +116,20 @@ export async function s3PutSignedUrl(
return url
}

export async function deleteUploadFile(upload: Upload) {
const fileKey = `uploads/${upload.agency.organizationId}/${upload.agencyId}/${upload.reportingPeriodId}/${upload.id}/${upload.filename}`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any necessity for a simple try / catch here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, will add as a follow-on PR!

@as1729 as1729 merged commit 7c1c7dc into main Jun 28, 2024
22 checks passed
@as1729 as1729 deleted the as/remove-test-files-from-production branch June 28, 2024 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@laurelkorwin laurelkorwin laurelkorwin approved these changes

@Vikariusu Vikariusu Awaiting requested review from Vikariusu

@vshia vshia Awaiting requested review from vshia

@jakekreider jakekreider Awaiting requested review from jakekreider

@TylerHendrickson TylerHendrickson Awaiting requested review from TylerHendrickson

Assignees
No one assigned
Labels
enhancement New feature or request javascript
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@as1729 @laurelkorwin