Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(tests): Add multiple exception support to EOF tests #759

Merged
merged 4 commits into from
Aug 27, 2024

Conversation

shemnon
Copy link
Contributor

@shemnon shemnon commented Aug 22, 2024

🗒️ Description

There are some tests that are impossible to express without causing
multiple validation exceptions. Clients should be free to expose any of
those exceptions. To support this pipe notation for expected exceptions
is plumed through the EOF tests and implemented in some tests.

🔗 Related Issues

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: All converted JSON/YML tests from ethereum/tests have been added to converted-ethereum-tests.txt.
  • Tests: A PR with removal of converted JSON/YML tests from ethereum/tests have been opened.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@marioevz
Copy link
Member

The EOFException base exception type already allows concatenating exceptions to allow the returned exception to be any of a given list, we could do that first because I'm worried that being too flexible is sometimes a bad thing when it comes to tests verification.

These concatenated exceptions are dumped into the fixtures in the form of a pipe-separated string:
E.g.

EOFException.INVALID_VERSION | EOFException.UNDEFINED_INSTRUCTION

is dumped into the fixture as:

"results": {
    "Prague": {
        "exception": "EOFException.INVALID_VERSION|EOFException.UNDEFINED_INSTRUCTION",
        "result": false
    }
}

Could this apply to the cases included in this PR?

@shemnon
Copy link
Contributor Author

shemnon commented Aug 22, 2024

I'll definitely look into this. There are many more error cases than are seen in this PR, particualrly in he area of invalid layouts.

There are some tests that are impossible to express without causing
multiple validation exceptions. Clients should be free to expose any of
those exceptions. To support this pipe notation for expected exceptions
is plumed through the EOF tests and implemented in some tests.

Signed-off-by: Danno Ferrin <[email protected]>
@shemnon shemnon force-pushed the eof/multiple_exceptions branch from 29d124c to b39f8dd Compare August 26, 2024 15:09
@shemnon shemnon changed the title feat(tests) Add "multiple exceptions" eof exception feat(tests) Add multiple exception support to eof tests Aug 26, 2024
@shemnon
Copy link
Contributor Author

shemnon commented Aug 26, 2024

The code using EOFExceptions was not expecting the instance or list nomenclature, so I re-booted the PR, and plumbed exception instance or list support through eof_test and updated all the tests Besu will report multiple exceptions for.

Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good so far, just a couple of minor comments. Thanks!

src/ethereum_test_specs/eof.py Show resolved Hide resolved
src/ethereum_test_specs/eof.py Outdated Show resolved Hide resolved
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple more comments to make tox happy.

src/ethereum_test_specs/eof.py Outdated Show resolved Hide resolved
src/ethereum_test_specs/eof.py Outdated Show resolved Hide resolved
shemnon and others added 2 commits August 26, 2024 17:38
@marioevz marioevz changed the title feat(tests) Add multiple exception support to eof tests feat(tests): Add multiple exception support to eof tests Aug 27, 2024
Copy link
Member

@marioevz marioevz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'll push a commit containing the changelog update and then will merge.

@marioevz marioevz changed the title feat(tests): Add multiple exception support to eof tests feat(tests): Add multiple exception support to EOF tests Aug 27, 2024
@marioevz marioevz merged commit 9886436 into ethereum:main Aug 27, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants