[Mega tracking] - Align Prysm to 0.8.2

[prestonvanloon]

Reference: https://github.com/ethereum/eth2.0-specs/releases/tag/v0.8.2
Tasks below copied from the changelog.

Bug fixes:

• correctly use bytes8 domain in bls in test generation ([hotfix] bump py_ecc==1.7.1 ethereum/consensus-specs#1300)
• fix start shard for compact committees root (Fix start shard for compact committees root ethereum/consensus-specs#1319)
• properly check attestation bitlist lengths (Check attestation bit lengths ethereum/consensus-specs#1317)
• ignore latest messages in fork choice prior to latest justified (Found by Cem Özer: Ignore older latest messages in attesting balance ethereum/consensus-specs#1306)

Other:

• test generation refactor (Implement proposed test organization and structure improvements ethereum/consensus-specs#1320) Skipped per [Mega tracking] - Align Prysm to 0.8.2 #3144 (comment)

I left out ethereum/consensus-specs#1283, ethereum/consensus-specs#1314, ethereum/consensus-specs#1308 as these seem to be typo or English clarification PRs.

[terencechain]

Working on "ignore latest messages in fork choice prior to latest justified"

[prestonvanloon]

New test vectors are here: https://github.com/ethereum/eth2.0-spec-tests/releases/tag/v0.8.2. Note that there are three test bundles instead of one. This will cause a bit of a dependency refactoring / reorganizing.

For SSZ, the test are split into 3 files for each scenario.

• value.yaml
• meta.yaml
• serialized.ssz

The yaml files are still hex encoded strings so they will need preprocessing in the forked repository. The upstream issue (ethereum/consensus-spec-tests#5) is still valid, unfortunately, for SSZ tests.

For the mainnet / minimal tests, the bundles have been split into mainnet.tar.gz and minimal.tar.gz with a copy of the yaml and ssz encoded values. Given that we must continue to preprocess ssz test vectors, I'd argue that we remove the ssz encoded values and continue preprocessing the yaml to convert hex strings into canonical yaml format representation for binary data. This approach is the most minimal impact to current spectest suite.

In spec tests, the paths have change to include longer paths. Example for minimal.tar.gz.

tests/minimal/phase0/epoch_processing/crosslinks/pyspec_tests/double_late_crosslink/pre.ssz
tests/minimal/phase0/epoch_processing/crosslinks/pyspec_tests/double_late_crosslink/post.yaml
tests/minimal/phase0/epoch_processing/crosslinks/pyspec_tests/double_late_crosslink/post.ssz
tests/minimal/phase0/epoch_processing/crosslinks/pyspec_tests/double_late_crosslink/pre.yaml

These were previously:

tests/epoch_processing/crosslinks/crosslinks_minimal.yaml
tests/epoch_processing/crosslinks/crosslinks_mainnet.yaml

In any case the following are required:

• Updating the dependency bundle (consolidation to a single bundle, preprocessing of yaml (if needed), removing ssz (if needed))
• Updating the file paths in test
• Updating tests to read multiple files for input / expected output
• Update data structures to match new test input / output values

[prestonvanloon]

Options for handling testnet refactor:

1. Skip updating tests for 0.8.2. There are no substantial requirements here anyway. We can defer this to a later time and potentially wait for a better alternative format.
2. Adopt the new yamls (preprocessed to replace hex strings with canonical base64).
3. Adopt ssz encoded binary files (still need to preprocess yaml for ssz tests).

[prestonvanloon]

Based on an offline discussion, we're opting to skip the test vector refactor (option 1) as there is little to no benefit at this time with a substantial change to the codebase.

[prestonvanloon]

We're now targeting all aspects of 0.8.2, including test format.

[rauljordan]

Completed