Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyspec-SSZ: lists-rework (enable static generalized indices) + fully python class based now. #1180

Merged
merged 75 commits into from
Jun 25, 2019
Merged
Changes from 1 commit
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
5ddfe34
Simplified SSZ impl
protolambda Jun 20, 2019
7c42324
Added get_container_type to get_zero_value
vbuterin Jun 14, 2019
8919f62
Update test_libs/pyspec/eth2spec/utils/ssz/ssz_impl.py
vbuterin Jun 14, 2019
d1ecfd5
typing improvements
protolambda Jun 20, 2019
b6cf809
more improvements, and implement new space-efficient merkleization wi…
protolambda Jun 20, 2019
cd5f59e
fix bytes value check, fix default-type checking
protolambda Jun 20, 2019
54a1fa9
Update test_libs/pyspec/eth2spec/utils/ssz/ssz_impl.py
protolambda Jun 15, 2019
3a9b1fb
Update test_libs/pyspec/eth2spec/utils/ssz/ssz_impl.py
protolambda Jun 15, 2019
82e7392
default method for container is recognized now
protolambda Jun 15, 2019
108410d
Change byte to explict class instead of newtype
protolambda Jun 15, 2019
4ebdcea
highly experimental typing
protolambda Jun 20, 2019
97025c5
start updating virtual sizes of lists
protolambda Jun 20, 2019
08e6f32
typing improvements, type testing
protolambda Jun 16, 2019
8bd2e87
bugfixes and typing improvements
protolambda Jun 16, 2019
0a43003
minor test improvements
protolambda Jun 17, 2019
b89183a
Update spec for new SSZ with list max length
protolambda Jun 20, 2019
4c2adcc
Update 0_beacon-chain.md
vbuterin Jun 17, 2019
73ba419
check virtual lengths, fix imports
protolambda Jun 18, 2019
8c6ddd5
container field coercion
protolambda Jun 18, 2019
4aefc07
list-rework type fixes
protolambda Jun 18, 2019
439e4d4
Build spec
protolambda Jun 20, 2019
c9747b6
improve build spec, get clean dependencies list
protolambda Jun 20, 2019
8344d50
update beacon chain doc, use new types, avoid List
protolambda Jun 20, 2019
4b4bf87
update shard doc, use new types, avoid List
protolambda Jun 18, 2019
5be0c57
fix linting + mypy
protolambda Jun 18, 2019
6f46c1d
fix typing in spec builder monkey patch
protolambda Jun 18, 2019
6b82e3f
Modifications from Vitalik, to enable SSZ Partials
protolambda Jun 20, 2019
5048b9e
temporary fix for phase-1 spec typing
protolambda Jun 19, 2019
a33c678
update ssz testing/debug utils
protolambda Jun 19, 2019
7cdec74
fix field iteration crash in ssz typing
protolambda Jun 19, 2019
4e747fb
fixes for class based ssz typing
protolambda Jun 20, 2019
977856b
ssz typing now subclasses list/bytes, much easier to work with than w…
protolambda Jun 20, 2019
82240d8
fix vector default type
protolambda Jun 20, 2019
f157745
resolve some remaining list-rework rebase details
protolambda Jun 20, 2019
224c98a
last() method, no negative index lookups
protolambda Jun 20, 2019
8c6d2b4
update ssz-pyssz decoder for fuzzing
protolambda Jun 20, 2019
8bd2048
improve type coercion; coerce between equal-length uint subclasses
protolambda Jun 20, 2019
b4ef672
deal with deepcopy modifying vector length from 0 to full length duri…
protolambda Jun 20, 2019
2d67717
fix linting issues + make spec builder remove comments in container r…
protolambda Jun 20, 2019
4dcfee2
remove unused spec-helper from spec builder
protolambda Jun 20, 2019
d8f470b
enable slicing of SSZ lists/vectors
protolambda Jun 20, 2019
6338c5b
fix custody bug, needs review from Carl
protolambda Jun 20, 2019
f27c44b
fix deposit negative index fail
protolambda Jun 20, 2019
c203724
comment out old deposit test, re-introduced soon maybe, cc Justin
protolambda Jun 20, 2019
3d8466f
make Bit check not use "is", and remove duplicate line
protolambda Jun 20, 2019
6648b3c
remove old deposits test, there is no deposit index in deposit data a…
protolambda Jun 20, 2019
e99c864
Deltas = NewType('Deltas', TypingList[Gwei])
hwwhww Jun 20, 2019
b7b2fee
uint add/sub type checking, fixes #1029
protolambda Jun 21, 2019
80c40f5
merge dev, resolve minor merge conflicts
protolambda Jun 21, 2019
d1fa3ac
remove unused dependency
protolambda Jun 21, 2019
d463ada
Merge branch 'dev' into list-rework
djrtwo Jun 21, 2019
c09e45c
fix rule_4 underflow and split out genesis finality test
djrtwo Jun 21, 2019
8f99741
remove commented old code
djrtwo Jun 21, 2019
0b0e9a5
Merge branch 'dev' into list-rework
protolambda Jun 21, 2019
16093eb
Merge branch 'dev' into list-rework
protolambda Jun 22, 2019
00aae07
type annotation clean up
protolambda Jun 22, 2019
f95e731
fix get_active_validator_indices typing usage
protolambda Jun 22, 2019
dd5ad2e
remove unnecessary (and now outdated) type hints, update List encodin…
protolambda Jun 22, 2019
e873bbd
support list casting
protolambda Jun 22, 2019
47034a6
fix imports in helper test file
protolambda Jun 22, 2019
0249cf6
fix lint, and update encoder to handle the few imported types well
protolambda Jun 22, 2019
9befe09
test merkleize chunks
protolambda Jun 22, 2019
da858f1
fix int encoding, fix list randomization size limit.
protolambda Jun 22, 2019
a5a2e29
remove unnecessary argument, typing is based on values fully now
protolambda Jun 22, 2019
1408a1e
undo tuple wrapping
protolambda Jun 22, 2019
1972cca
Merge branch 'dev' into list-rework
djrtwo Jun 24, 2019
82ae180
clean up list limit constants
protolambda Jun 24, 2019
c73417b
deserialize-basic detail, make subclass
protolambda Jun 24, 2019
5989e5c
use Bool as base name, make Bit an alias
protolambda Jun 24, 2019
8b88c3f
Merge remote-tracking branch 'origin' into list-rework
protolambda Jun 24, 2019
81a2c84
Merge branch 'list-rework' of https://github.com/ethereum/eth2.0-spec…
protolambda Jun 24, 2019
9fb5806
be explicit about input for balance sum
protolambda Jun 24, 2019
a5b7564
hash-tree-root tests
protolambda Jun 25, 2019
45dbf5a
Remove old Deltas reference
protolambda Jun 25, 2019
054a157
get rid of TypingList, add MutableSequence
protolambda Jun 25, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions scripts/build_spec.py
Original file line number Diff line number Diff line change
@@ -12,8 +12,7 @@


PHASE0_IMPORTS = '''from typing import (
Any, Callable, Iterable, Dict, Set, Tuple, NewType,
List as TypingList,
Any, Callable, Iterable, Dict, Set, Sequence, Tuple,
protolambda marked this conversation as resolved.
Show resolved Hide resolved
)

from dataclasses import (
@@ -38,7 +37,7 @@
from eth2spec.utils.hash_function import hash
'''
PHASE1_IMPORTS = '''from typing import (
Any, Callable, Dict, Optional, Set, Tuple, Iterable, NewType,
Any, Callable, Dict, Iterable, Optional, Set, Sequence, Tuple,
protolambda marked this conversation as resolved.
Show resolved Hide resolved
List as TypingList
)

@@ -79,13 +78,13 @@ def hash(x: bytes) -> Hash:

# Monkey patch validator compute committee code
_compute_committee = compute_committee
committee_cache: Dict[Tuple[Hash, Hash, int, int], Tuple[ValidatorIndex, ...]] = {}
committee_cache: Dict[Tuple[Hash, Hash, int, int], Sequence[ValidatorIndex]] = {}


def compute_committee(indices: Tuple[ValidatorIndex, ...], # type: ignore
def compute_committee(indices: Sequence[ValidatorIndex], # type: ignore
seed: Hash,
index: int,
count: int) -> Tuple[ValidatorIndex, ...]:
count: int) -> Sequence[ValidatorIndex]:
param_hash = (hash(b''.join(index.to_bytes(length=4, byteorder='little') for index in indices)), seed, index, count)

if param_hash not in committee_cache:
@@ -154,7 +153,6 @@ def objects_to_spec(functions: Dict[str, str],
spec = (
imports
+ '\n\n' + new_type_definitions
+ '\n\n' + "Deltas = NewType('Deltas', TypingList[Gwei])"
+ '\n\n' + constants_spec
+ '\n\n\n' + ssz_objects_instantiation_spec
+ '\n\n' + functions_spec
64 changes: 32 additions & 32 deletions specs/core/0_beacon-chain.md
Original file line number Diff line number Diff line change
@@ -651,13 +651,11 @@ def is_slashable_validator(validator: Validator, epoch: Epoch) -> bool:
### `get_active_validator_indices`

```python
def get_active_validator_indices(state: BeaconState, epoch: Epoch) -> List[ValidatorIndex, VALIDATOR_REGISTRY_SIZE]:
def get_active_validator_indices(state: BeaconState, epoch: Epoch) -> Sequence[ValidatorIndex]:
"""
Get active validator indices at ``epoch``.
"""
return List[ValidatorIndex, VALIDATOR_REGISTRY_SIZE](
i for i, v in enumerate(state.validators) if is_active_validator(v, epoch)
)
return [ValidatorIndex(i) for i, v in enumerate(state.validators) if is_active_validator(v, epoch)]
```

### `increase_balance`
@@ -819,7 +817,7 @@ def get_beacon_proposer_index(state: BeaconState) -> ValidatorIndex:
### `verify_merkle_branch`

```python
def verify_merkle_branch(leaf: Hash, proof: Tuple[Hash, ...], depth: int, index: int, root: Hash) -> bool:
def verify_merkle_branch(leaf: Hash, proof: Sequence[Hash], depth: int, index: int, root: Hash) -> bool:
"""
Verify that the given ``leaf`` is on the merkle branch ``proof``
starting with the given ``root``.
@@ -863,19 +861,19 @@ def get_shuffled_index(index: ValidatorIndex, index_count: int, seed: Hash) -> V
### `compute_committee`

```python
def compute_committee(indices: Tuple[ValidatorIndex, ...],
seed: Hash, index: int, count: int) -> Tuple[ValidatorIndex, ...]:
def compute_committee(indices: Sequence[ValidatorIndex],
seed: Hash, index: int, count: int) -> Sequence[ValidatorIndex]:
start = (len(indices) * index) // count
end = (len(indices) * (index + 1)) // count
return tuple(indices[get_shuffled_index(ValidatorIndex(i), len(indices), seed)] for i in range(start, end))
return [indices[get_shuffled_index(ValidatorIndex(i), len(indices), seed)] for i in range(start, end)]
```

### `get_crosslink_committee`

```python
def get_crosslink_committee(state: BeaconState, epoch: Epoch, shard: Shard) -> Tuple[ValidatorIndex, ...]:
def get_crosslink_committee(state: BeaconState, epoch: Epoch, shard: Shard) -> Sequence[ValidatorIndex]:
return compute_committee(
indices=tuple(get_active_validator_indices(state, epoch)),
indices=get_active_validator_indices(state, epoch),
seed=generate_seed(state, epoch),
index=(shard + SHARD_COUNT - get_epoch_start_shard(state, epoch)) % SHARD_COUNT,
count=get_epoch_committee_count(state, epoch),
@@ -887,13 +885,13 @@ def get_crosslink_committee(state: BeaconState, epoch: Epoch, shard: Shard) -> T
```python
def get_attesting_indices(state: BeaconState,
attestation_data: AttestationData,
bitfield: bytes) -> Tuple[ValidatorIndex, ...]:
bitfield: bytes) -> Sequence[ValidatorIndex]:
"""
Return the sorted attesting indices corresponding to ``attestation_data`` and ``bitfield``.
"""
committee = get_crosslink_committee(state, attestation_data.target_epoch, attestation_data.crosslink.shard)
assert verify_bitfield(bitfield, len(committee))
return tuple(sorted([index for i, index in enumerate(committee) if get_bitfield_bit(bitfield, i) == 0b1]))
return sorted([index for i, index in enumerate(committee) if get_bitfield_bit(bitfield, i) == 0b1])
```

### `int_to_bytes`
@@ -1139,7 +1137,7 @@ def slash_validator(state: BeaconState,

### Genesis trigger

Before genesis has been triggered and whenever the deposit contract emits a `Deposit` log, call the function `is_genesis_trigger(deposits: Iterable[Deposit], timestamp: uint64) -> bool` where:
Before genesis has been triggered and whenever the deposit contract emits a `Deposit` log, call the function `is_genesis_trigger(deposits: Sequence[Deposit], timestamp: uint64) -> bool` where:

* `deposits` is the list of all deposits, ordered chronologically, up to and including the deposit triggering the latest `Deposit` log
* `timestamp` is the Unix timestamp in the Ethereum 1.0 block that emitted the latest `Deposit` log
@@ -1156,7 +1154,7 @@ When `is_genesis_trigger(deposits, timestamp) is True` for the first time let:
*Note*: The function `is_genesis_trigger` has yet to be agreed by the community, and can be updated as necessary. We define the following testing placeholder:

```python
def is_genesis_trigger(deposits: Iterable[Deposit], timestamp: uint64) -> bool:
def is_genesis_trigger(deposits: Sequence[Deposit], timestamp: uint64) -> bool:
# Process deposits
state = BeaconState()
for deposit in deposits:
@@ -1178,7 +1176,7 @@ def is_genesis_trigger(deposits: Iterable[Deposit], timestamp: uint64) -> bool:
Let `genesis_state = get_genesis_beacon_state(genesis_deposits, genesis_time, genesis_eth1_data)`.

```python
def get_genesis_beacon_state(deposits: Iterable[Deposit], genesis_time: int, eth1_data: Eth1Data) -> BeaconState:
def get_genesis_beacon_state(deposits: Sequence[Deposit], genesis_time: int, eth1_data: Eth1Data) -> BeaconState:
state = BeaconState(
genesis_time=genesis_time,
eth1_data=eth1_data,
@@ -1195,8 +1193,10 @@ def get_genesis_beacon_state(deposits: Iterable[Deposit], genesis_time: int, eth
validator.activation_eligibility_epoch = GENESIS_EPOCH
validator.activation_epoch = GENESIS_EPOCH

# Populate active_index_roots
genesis_active_index_root = hash_tree_root(get_active_validator_indices(state, GENESIS_EPOCH))
# Populate active_index_roots
genesis_active_index_root = hash_tree_root(
List[ValidatorIndex, VALIDATOR_REGISTRY_SIZE](get_active_validator_indices(state, GENESIS_EPOCH))
protolambda marked this conversation as resolved.
Show resolved Hide resolved
)
for index in range(EPOCHS_PER_HISTORICAL_VECTOR):
state.active_index_roots[index] = genesis_active_index_root

@@ -1271,25 +1271,25 @@ def process_epoch(state: BeaconState) -> None:

```python
def get_total_active_balance(state: BeaconState) -> Gwei:
return get_total_balance(state, get_active_validator_indices(state, get_current_epoch(state)))
return get_total_balance(state, set(get_active_validator_indices(state, get_current_epoch(state))))
protolambda marked this conversation as resolved.
Show resolved Hide resolved
```

```python
def get_matching_source_attestations(state: BeaconState, epoch: Epoch) -> Iterable[PendingAttestation]:
def get_matching_source_attestations(state: BeaconState, epoch: Epoch) -> Sequence[PendingAttestation]:
assert epoch in (get_current_epoch(state), get_previous_epoch(state))
return state.current_epoch_attestations if epoch == get_current_epoch(state) else state.previous_epoch_attestations
```

```python
def get_matching_target_attestations(state: BeaconState, epoch: Epoch) -> Iterable[PendingAttestation]:
def get_matching_target_attestations(state: BeaconState, epoch: Epoch) -> Sequence[PendingAttestation]:
return [
a for a in get_matching_source_attestations(state, epoch)
if a.data.target_root == get_block_root(state, epoch)
]
```

```python
def get_matching_head_attestations(state: BeaconState, epoch: Epoch) -> Iterable[PendingAttestation]:
def get_matching_head_attestations(state: BeaconState, epoch: Epoch) -> Sequence[PendingAttestation]:
return [
a for a in get_matching_source_attestations(state, epoch)
if a.data.beacon_block_root == get_block_root_at_slot(state, get_attestation_data_slot(state, a.data))
@@ -1298,22 +1298,22 @@ def get_matching_head_attestations(state: BeaconState, epoch: Epoch) -> Iterable

```python
def get_unslashed_attesting_indices(state: BeaconState,
attestations: Iterable[PendingAttestation]) -> Iterable[ValidatorIndex]:
attestations: Sequence[PendingAttestation]) -> Set[ValidatorIndex]:
output = set() # type: Set[ValidatorIndex]
for a in attestations:
output = output.union(get_attesting_indices(state, a.data, a.aggregation_bitfield))
return sorted(filter(lambda index: not state.validators[index].slashed, list(output)))
return set(filter(lambda index: not state.validators[index].slashed, list(output)))
```

```python
def get_attesting_balance(state: BeaconState, attestations: Iterable[PendingAttestation]) -> Gwei:
def get_attesting_balance(state: BeaconState, attestations: Sequence[PendingAttestation]) -> Gwei:
return get_total_balance(state, get_unslashed_attesting_indices(state, attestations))
```

```python
def get_winning_crosslink_and_attesting_indices(state: BeaconState,
epoch: Epoch,
shard: Shard) -> Tuple[Crosslink, Iterable[ValidatorIndex]]:
shard: Shard) -> Tuple[Crosslink, Set[ValidatorIndex]]:
attestations = [a for a in get_matching_source_attestations(state, epoch) if a.data.crosslink.shard == shard]
crosslinks = list(filter(
lambda c: hash_tree_root(state.current_crosslinks[shard]) in (c.parent_root, hash_tree_root(c)),
@@ -1402,11 +1402,11 @@ def get_base_reward(state: BeaconState, index: ValidatorIndex) -> Gwei:
```

```python
def get_attestation_deltas(state: BeaconState) -> Tuple[Deltas, Deltas]:
def get_attestation_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]:
previous_epoch = get_previous_epoch(state)
total_balance = get_total_active_balance(state)
rewards = Deltas([Gwei(0) for _ in range(len(state.validators))])
penalties = Deltas([Gwei(0) for _ in range(len(state.validators))])
rewards = [Gwei(0) for _ in range(len(state.validators))]
penalties = [Gwei(0) for _ in range(len(state.validators))]
eligible_validator_indices = [
ValidatorIndex(index) for index, v in enumerate(state.validators)
if is_active_validator(v, previous_epoch) or (v.slashed and previous_epoch + 1 < v.withdrawable_epoch)
@@ -1453,9 +1453,9 @@ def get_attestation_deltas(state: BeaconState) -> Tuple[Deltas, Deltas]:
```

```python
def get_crosslink_deltas(state: BeaconState) -> Tuple[Deltas, Deltas]:
rewards = Deltas([Gwei(0) for _ in range(len(state.validators))])
penalties = Deltas([Gwei(0) for _ in range(len(state.validators))])
def get_crosslink_deltas(state: BeaconState) -> Tuple[Sequence[Gwei], Sequence[Gwei]]:
rewards = [Gwei(0) for _ in range(len(state.validators))]
penalties = [Gwei(0) for _ in range(len(state.validators))]
epoch = get_previous_epoch(state)
for offset in range(get_epoch_committee_count(state, epoch)):
shard = Shard((get_epoch_start_shard(state, epoch) + offset) % SHARD_COUNT)
@@ -1642,7 +1642,7 @@ def process_operations(state: BeaconState, body: BeaconBlockBody) -> None:
(body.deposits, process_deposit),
(body.voluntary_exits, process_voluntary_exit),
(body.transfers, process_transfer),
) # type: Tuple[Tuple[List, Callable], ...]
) # type: Sequence[Tuple[List, Callable]]
for operations, function in all_operations:
for operation in operations:
function(state, operation)
18 changes: 9 additions & 9 deletions specs/core/1_shard-data-chains.md
Original file line number Diff line number Diff line change
@@ -133,7 +133,7 @@ def get_period_committee(state: BeaconState,
epoch: Epoch,
shard: Shard,
index: int,
count: int) -> Tuple[ValidatorIndex, ...]:
count: int) -> Sequence[ValidatorIndex]:
"""
Return committee for a period. Used to construct persistent committees.
"""
@@ -159,7 +159,7 @@ def get_switchover_epoch(state: BeaconState, epoch: Epoch, index: ValidatorIndex
```python
def get_persistent_committee(state: BeaconState,
shard: Shard,
slot: Slot) -> Tuple[ValidatorIndex, ...]:
slot: Slot) -> Sequence[ValidatorIndex]:
"""
Return the persistent committee for the given ``shard`` at the given ``slot``.
"""
@@ -193,7 +193,7 @@ def get_shard_proposer_index(state: BeaconState,
shard: Shard,
slot: Slot) -> Optional[ValidatorIndex]:
# Randomly shift persistent committee
persistent_committee = get_persistent_committee(state, shard, slot)
persistent_committee = list(get_persistent_committee(state, shard, slot))
seed = hash(state.current_shuffling_seed + int_to_bytes(shard, length=8) + int_to_bytes(slot, length=8))
random_index = bytes_to_int(seed[0:8]) % len(persistent_committee)
persistent_committee = persistent_committee[random_index:] + persistent_committee[:random_index]
@@ -248,7 +248,7 @@ def verify_shard_attestation_signature(state: BeaconState,
### `compute_crosslink_data_root`

```python
def compute_crosslink_data_root(blocks: Iterable[ShardBlock]) -> Bytes32:
def compute_crosslink_data_root(blocks: Sequence[ShardBlock]) -> Bytes32:
def is_power_of_two(value: int) -> bool:
return (value > 0) and (value & (value - 1) == 0)

@@ -287,9 +287,9 @@ Let:
* `candidate` be a candidate `ShardBlock` for which validity is to be determined by running `is_valid_shard_block`

```python
def is_valid_shard_block(beacon_blocks: TypingList[BeaconBlock],
def is_valid_shard_block(beacon_blocks: Sequence[BeaconBlock],
beacon_state: BeaconState,
valid_shard_blocks: Iterable[ShardBlock],
valid_shard_blocks: Sequence[ShardBlock],
candidate: ShardBlock) -> bool:
# Check if block is already determined valid
for _, block in enumerate(valid_shard_blocks):
@@ -336,7 +336,7 @@ def is_valid_shard_block(beacon_blocks: TypingList[BeaconBlock],
assert proposer_index is not None
assert bls_verify(
pubkey=beacon_state.validators[proposer_index].pubkey,
message_hash=signing_root(block),
message_hash=signing_root(candidate),
signature=candidate.signature,
domain=get_domain(beacon_state, DOMAIN_SHARD_PROPOSER, slot_to_epoch(candidate.slot)),
)
@@ -353,7 +353,7 @@ Let:
* `candidate` be a candidate `ShardAttestation` for which validity is to be determined by running `is_valid_shard_attestation`

```python
def is_valid_shard_attestation(valid_shard_blocks: Iterable[ShardBlock],
def is_valid_shard_attestation(valid_shard_blocks: Sequence[ShardBlock],
beacon_state: BeaconState,
candidate: ShardAttestation) -> bool:
# Check shard block
@@ -383,7 +383,7 @@ Let:

```python
def is_valid_beacon_attestation(shard: Shard,
shard_blocks: TypingList[ShardBlock],
shard_blocks: Sequence[ShardBlock],
beacon_state: BeaconState,
valid_attestations: Set[Attestation],
protolambda marked this conversation as resolved.
Show resolved Hide resolved
candidate: Attestation) -> bool: