Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): Add --check-return to execute #6765

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

Conversation

aakoshh
Copy link
Contributor

@aakoshh aakoshh commented Dec 10, 2024

Description

Problem*

Followup to #6757
Related to #6516

Summary*

Adds a --check-return option to the execute command; if true and there is a return entry in Prover.toml, its value is compared to the ABI encoded result returned from the execution, the one printed to stdout by the command upon successful execution. If the two don't match, an error is returned, the two values are printed, and the program exits with an error code.

This is done after the witness is saved, so we can inspect what went into the file if necessary; the main goal is to be able to use this integration testing, since the ABI parser already looks for the return key.

build.rs has been amended to use the --check-return flag when running the tests in execution_success, so we can verify the expected output. Added the test case from #6516 as extra coverage in addition to the unit test in #6757

The execute command is also doing a check now to demand that if the circuit, according to its ABI, should have a return value, then that return value is Some value, to help us catch cases where some witnesses have not been assigned, even if we did not configure their expected values in tests.

Additional Context

Example of running the test with wrong output in Prover.toml:

cargo test -q -p nargo_cli --test execute return_twice::forcebrillig_false_inliner_0

running 1 test
F
failures:

---- tests::execution_success::test_return_twice::forcebrillig_false_inliner_0_expects stdout ----
thread 'tests::execution_success::test_return_twice::forcebrillig_false_inliner_0_expects' panicked at /rustc/a28077b28a02b92985b3a3faecf92813155f1ea1/library/core/src/ops/function.rs:250:5:
Unexpected failure.
code=1
stderr=```"Unexpected return value: expected Vec([Field(100), Field(20)]); got Some(Vec([Field(100), Field(100)]))\n"```
command=`"/Users/aakoshh/Work/aztec/noir/target/debug/nargo" "--program-dir" "/Users/aakoshh/Work/aztec/noir/test_programs/execution_success/return_twice" "execute" "--force" "--inliner-aggressiveness" "0" "--check-return"`
code=1
stdout=
[return_twice] Circuit witness successfully solved
[return_twice] Circuit output: Vec([Field(100), Field(100)])
[return_twice] Witness saved to /Users/aakoshh/Work/aztec/noir/test_programs/execution_success/return_twice/target/return_twice.gz


stderr="Unexpected return value: expected Vec([Field(100), Field(20)]); got Some(Vec([Field(100), Field(100)]))\n"

note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::execution_success::test_return_twice::forcebrillig_false_inliner_0_expects

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1664 filtered out; finished in 0.59s

error: test failed, to rerun pass `-p nargo_cli --test execute`

Documentation*

Check one:

  • No documentation needed.
  • Documentation included in this PR.
  • [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

  • I have tested the changes locally.
  • I have formatted the changes with Prettier and/or cargo fmt on default settings.

@aakoshh aakoshh requested a review from a team December 10, 2024 20:27
Copy link
Contributor

github-actions bot commented Dec 10, 2024

Compilation Sample

Program Compilation Time %
sha256_regression 0m1.511s -1%
regression_4709 0m1.561s -3%
ram_blowup_regression 0m17.084s -1%

Copy link
Contributor

github-actions bot commented Dec 10, 2024

Peak Memory Sample

Program Peak Memory %
keccak256 81.38M 0%
workspace 121.76M -1%
regression_4709 333.59M 0%
ram_blowup_regression 2.55G 0%

@aakoshh aakoshh changed the title feat(exec): Add --check-return to execute and verify in integration tests feat(exec): Add --check-return to execute, and verify in integration tests Dec 10, 2024
@aakoshh aakoshh changed the title feat(exec): Add --check-return to execute, and verify in integration tests feat(cli): Add --check-return to execute Dec 10, 2024
@TomAFrench
Copy link
Member

Is this reading necessary to be honest so it's kind of surprised that we didn't error out when ABI decoding already.

If we have a witness map from which we want to decode return values and we find empty witnesses at indices that should hold return values then we should just error and that should avoid the need for this flag I think.

@aakoshh
Copy link
Contributor Author

aakoshh commented Dec 10, 2024

@TomAFrench the reason it did not complain about the gaps in the witness map is explained here:

Unlike for the circuit inputs, we tolerate not being able to find the witness values for the return value.
This is because the user may be decoding a partial witness map for which is hasn't been calculated yet.
If a return value is expected, this should be checked for by the user.

You are right that based on the ABI we can tell if the circuit has any return values. I'm not sure if execute can always demand that abi.decode returns Some; it sounds like fair assumption in the context of the execute command 🤷

But I'd argue that the flag still has value to easily assert the actual value of what is being returned, not just that something is, at least in the context of integration testing, where we use Prover.toml for inputs and it readily handles the output as well. At least it can catch inconsistencies like in the diamond_deps example.

@TomAFrench
Copy link
Member

Ok, sorry I didn't read fully as I was on the tube. I'll have a look at this properly tomorrow.

@aakoshh
Copy link
Contributor Author

aakoshh commented Dec 10, 2024

I added another check to see if the return value is empty when it should be non-empty according to the ABI; all the tests are passing so I suppose that means this is a reasonable expectation. Again doing it after saving the (partial) witness map to file, but could be one earlier if the value cannot be printed.

Maybe we could compare return in Prover.toml whenever it is present, without any flags?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants