Add functional smoke tests and automated testing workflow

[mitchelbaker-cisa]

🗣 Description

Added smoke tests (/Testing/Functional/SmokeTests/smoke_test.py) to verify the following:

• scubagoggles output (all files/directories are created)
• ScubaResults.json (determine if there are API exceptions or other errors produced)
• Parent/Individual html reports (navigation links, agency data, baseline report data, and policy tables display as expected)

Added an automated testing workflow (.github/workflows/run_smoke_test.py) that runs the smoke tests against Windows/macOS and across different python versions. The workflow runs when a PR is opened, reopened, synchronized (a new commit is added), and when a commit is pushed into main.

The workflow has been tested against Python versions ranging from 3.9 - 3.12. Due to limitations with the numpy package testing is limited < v3.9. (numpy v1.24.4 is the highest supported version for Python v3.8.4 so there's issues with downgrading to numpy v1.26.4)

💭 Motivation and context

We manually test scubagoggles output, python min/max supported versions, etc. which is resource intensive. The purpose of these smoke tests and automated workflows is to ease the burden of testing when new PRs are opened, changes are added into main, and prior to releases.

closes #171

Create future todos:

• Use Selenium to check for the correct baseline and tool version in the agency table (e.g. 0.2, 0.2.0) - Check for the correct baseline version in generated reports when running smoke tests #339
• Update run_smoke_test.py to take in input when running the workflow manually. This will allow our developers to input specific Python versions as opposed to hardcoded values. - Update run_smoke_test.yml with ability to intake user inputs from workflow dispatch  #340
• Update docs on how to run smoke tests - Update documentation on how to run smoke tests #341

🧪 Testing

pip install pytest selenium

A service account is required to run the smoke tests. Replace "[email protected]" with your subject email, then replace "[email protected]" with the domain you're running scubagoggles against.

pytest -s -vvv ./Testing/Functional/SmokeTests/ --subjectemail="[email protected]" --domain="domain.com"

✅ Pre-approval checklist

• This PR has an informative and human-readable title.
• Changes are limited to a single goal - eschew scope creep!
• If applicable, All future TODOs are captured in issues, which are referenced in the PR description.
• The relevant issues PR resolves are linked preferably via closing keywords.
• All relevant type-of-change labels have been added.
• I have read and agree to the CONTRIBUTING.md document.
• These code changes follow cisagov code standards.
• All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
• Tests have been added and/or modified to cover the changes in this PR.
• All new and existing tests pass.

✅ Pre-merge Checklist

• This PR has been smoke tested to ensure main is in a functional state when this PR is merged.
• Squash all commits into one PR level commit using the Squash and merge button.

✅ Post-merge Checklist

• Delete the branch to clean up.
• Close issues resolved by this PR if the closing keywords did not activate.

[LaurenBassett]

That did it! I have it running successfully on Mac now. I guess I got a little confused with the service account vs. domain vs. admin account.

Awesome! Thanks for documenting the setup process, a lot of good feedback here. Are there areas which can be more clear, perhaps changing the domain parameter to customerdomain?

I think there are a few bits that would be helpful in the documentation:

• I was unclear what subject email was referring to; my scuba domain email address, or the service account's email address.
• Having the example be [email protected], or even [email protected] may be clearer
• I'd make a stronger note about the need for a service account and to ensure credentials are updated
• The instructions for what to replace are confusing:

A service account is required to run the smoke tests. Replace "[email protected]" with your subject email, then replace "[email protected]" with the domain you're running scubagoggles against.

• I was using [email protected] for the domain, not the basic domain, which was causing that last error