Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a conformance test workflow #2670

Merged
merged 2 commits into from
Jul 11, 2024

Conversation

jku
Copy link
Member

@jku jku commented Jul 10, 2024

  • The conformance test suite is likely to still change quite a bit so the workflow is not enabled on PRs yet (and does not pin the action version)
  • The actual conformance client is copied from the tuf-conformance project
  • This is mostly a test to see how things should work out, and a demonstration of how the tuf-conformance project should be used

* The conformance test suite is likely to still change quite a bit so
  the workflow is not enabled on PRs yet
* The actual conformance client is copied from the tuf-conformance project
* This is mostly a test to see how things should work out, and a
  demonstration of how the tuf-conformance project should be used

Signed-off-by: Jussi Kukkonen <[email protected]>
@jku jku force-pushed the add-conformance-workflow branch from 3f49d7d to 0b85ed5 Compare July 10, 2024 13:15
@jku

This comment was marked as outdated.

@coveralls
Copy link

coveralls commented Jul 10, 2024

Pull Request Test Coverage Report for Build 9894255152

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.805%

Totals Coverage Status
Change from base Build 9858741524: 0.0%
Covered Lines: 1554
Relevant Lines: 1591

💛 - Coveralls

@jku
Copy link
Member Author

jku commented Jul 10, 2024

There is no automated way to run he conformance test suite locally (using your local python-tuf version) but we could potentially add something into tox... it will need a bit of thinking though.

It's not too difficult manually, something like this should work:

pip install -e $CONFORMANCE_SUITE_DIR -e $PYTHON_TUF_DIR
pytest "$CONFORMANCE_SUITE_DIR/tuf_conformance" --entrypoint $PYTHON_TUF_DIR/.github/scripts/conformance-client.py

@jku jku marked this pull request as ready for review July 10, 2024 14:02
@jku jku requested a review from lukpueh July 10, 2024 14:07
Copy link
Member

@lukpueh lukpueh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool. Tried it locally and it worked nicely. Does the action also work with tuf-confromance@main? AFAICS you tried it against a patched action?
https://github.com/jku/python-tuf/actions/runs/9874869528

conformance:
runs-on: ubuntu-latest
steps:
- name: Checkout the client wrapper
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: We don't call it wrapper elsewhere here. IMO calling it conformance client would be clearer.

@jku
Copy link
Member Author

jku commented Jul 11, 2024

Yeah I used the patched branch for testing but that branch has since been merged -- main should work right now (obviously it will break at some point so we should start pinning the action as soon as we have something that looks like a release)

@jku jku merged commit cb550d2 into theupdateframework:develop Jul 11, 2024
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants