change(ci): Generate mainnet checkpoints in CI

[teor2345]

Motivation

In #6367 we want to generate checkpoints in CI. This is the mainnet CI and Rust test harness part of that ticket.

Complex Code or Requirements

This is just test code.

Solution

CI workflow:

• Add Mainnet CI workflow for zebra-checkpoints

Rust test harness:

• Add Mainnet and Testnet zebra-checkpoints test harnesses
• Build zebra-utils binaries for zebrad integration tests
• Start generating checkpoints from the last compiled-in checkpoint
• Try zebra-checkpoints in the system path first, then the cargo path

Docker test harness:

• Add zebra-checkpoints test entrypoints
• Add documentation for how to run these tests individually

zebra-checkpoints binary:

• Add zebra-checkpoints to test docker images
• Enable zebra-checkpoints feature in the test image

Logging:

• Silence zebrad logs while generating checkpoints
• Log a startup message to standard error when launching zebra-checkpoints

Related changes:

• Use the same features for (almost) all the docker tests
• Log the original test command name when showing stdout and stderr

And a bunch of minor fixes.

Review

This is a routine CI change.

Reviewer Checklist

• Will the PR name make sense to users?
  • Does it need extra CHANGELOG info? (new features, breaking changes, large changes)
• Are the PR labels correct?
• Does the code do what the ticket and PR says?
  • Does it change concurrent code, unsafe code, or consensus rules?
• How do you know it works? Does it have tests?

Follow Up Work

• Admin: Require the new mainnet job to pass before a PR can merge

Next PR:

• Add a workflow for testnet

[codecov]

Codecov Report

Merging #6550 (c311186) into main (d3ce022) will decrease coverage by 0.07%.
The diff coverage is 77.77%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6550      +/-   ##
==========================================
- Coverage   77.96%   77.89%   -0.07%     
==========================================
  Files         307      307              
  Lines       40261    40273      +12     
==========================================
- Hits        31388    31372      -16     
- Misses       8873     8901      +28     

[teor2345]

This works locally for me, using the same entrypoint.sh script.

There have been some weird issues where there's no zebra-checkpoints process output. I've seen it occasionally locally, and a lot in CI. They are a bit hard to diagnose, because the process doesn't fail. It just runs and exits, without any other output.

I've just pushed a workaround.

It should look something like:

[teor2345]

This is really strange - the test harness works locally, but in Docker + Google Cloud, zebra-checkpoints seems to immediately close its standard error as soon as it launches. (Or possibly never launch?)

Fri, 21 Apr 2023 07:29:09 GMT 2023-04-21T07:29:09.014654Z  INFO spawn_zebra_checkpoints_direct{network=Mainnet test_type=UpdateZebraCachedStateWithRpc zebrad_rpc_address=127.0.0.1:36529 last_checkpoint="2056451"}: Trying to launch zebra-checkpoints from cargo path... zebra_checkpoints_path="/opt/zebrad/target/release/zebra-checkpoints" args=Arguments({"--transport": Some("direct"), "--addr": Some("127.0.0.1:36529"), "--last-checkpoint": Some("2056451")})
Fri, 21 Apr 2023 07:29:09 GMT
Fri, 21 Apr 2023 07:29:09 GMT Child Stderr:
Fri, 21 Apr 2023 07:29:09 GMT [sentry] initialized disabled sentry client due to disabled or invalid DSN
Fri, 21 Apr 2023 07:29:09 GMT
Fri, 21 Apr 2023 07:29:09 GMT Error: 
Fri, 21 Apr 2023 07:29:09 GMT    0: stderr of command did not log any matches for the given regex,
Fri, 21 Apr 2023 07:29:09 GMT       within the 11h command timeout
Fri, 21 Apr 2023 07:29:09 GMT
Fri, 21 Apr 2023 07:29:09 GMT Location:
Fri, 21 Apr 2023 07:29:09 GMT    /opt/zebrad/zebra-test/src/command.rs:923
Fri, 21 Apr 2023 07:29:09 GMT
Fri, 21 Apr 2023 07:29:09 GMT Match Regex:
Fri, 21 Apr 2023 07:29:09 GMT    [
Fri, 21 Apr 2023 07:29:09 GMT        "calculating checkpoints",
Fri, 21 Apr 2023 07:29:09 GMT    ]
Fri, 21 Apr 2023 07:29:09 GMT
Fri, 21 Apr 2023 07:29:09 GMT Command:
Fri, 21 Apr 2023 07:29:09 GMT    "/opt/zebrad/target/release/zebra-checkpoints" "--transport" "direct" "--addr" "127.0.0.1:36529" "--last-checkpoint" "2056451"

https://github.com/ZcashFoundation/zebra/actions/runs/4762133164/jobs/8464755724?pr=6550#step:8:218

[teor2345]

When I run zebra-checkpoints --help locally, it gives me usage info. But when I run it in CI I get no output:

2023-04-24T01:28:34.846314Z INFO spawn_zebra_checkpoints_direct{network=Mainnet test_type=UpdateZebraCachedStateWithRpc zebrad_rpc_address=127.0.0.1:41547 last_checkpoint="2056451"}: Trying to launch zebra-checkpoints by searching system $PATH... zebra_checkpoints_path="zebra-checkpoints" args=Arguments({"--transport": Some("direct"), "--addr": Some("127.0.0.1:41547"), "--last-checkpoint": Some("2056451")}) temp_dir="/tmp/zebrad_testsQtGqof" system_path=Ok("/opt/zebrad/.cargo/bin:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin") usr_local_zebra_checkpoints_info=Ok(Metadata { file_type: FileType(FileType { mode: 33261 }), is_dir: false, is_file: true, permissions: Permissions(FilePermissions { mode: 33261 }), modified: Ok(SystemTime { tv_sec: 1682299330, tv_nsec: 0 }), accessed: Ok(SystemTime { tv_sec: 1682299330, tv_nsec: 0 }), created: Ok(SystemTime { tv_sec: 1682299613, tv_nsec: 482256533 }), .. })

(usage output should be here)

https://github.com/ZcashFoundation/zebra/actions/runs/4781308585/jobs/8500061799#step:8:179

It seems like there's something wrong with the zebra-checkpoints binary, or with the test harness. I'll run the test Docker image locally to find out.

[teor2345]

@Mergifyio refresh

[mergify]

refresh

✅ Pull request refreshed

[teor2345]

Anyone can re-approve this PR, I just rebased it after PR #6516 merged.

[gustavovalverde]

Re-approving