Pythonize epoch transitions

[vbuterin]

This is first and foremost a refactor of the epoch transition to define it entirely using python code.

To make the code executable, a number of substantive changes were also included. See below for a detailed change log of such changes.

Substantive changes

• Merkleization friendly pending attestations (Merkleisation friendly pending attestations #697)
  • Motivation
    • cleaner selection of attestations from previous and current epoch
    • merkleization friendly when cleaning up attestations at end of epoch transition
  • Changelog
    • remove latest_attestations from BeaconState
    • add previous_epoch_attestations and current_epoch_attestations to BeaconState
    • when processing attestations from block, append to previous/current epoch attestations array depending on if attestation.data.slot is from previous or current epoch
    • at end of epoch transition clean up attestations via the following:
      • Set state.previous_epoch_attestations = state.current_epoch_attestations.
      • Set state.current_epoch_attestations = [].
• parent root reconstruction + granular state roots + body/header segregation (Parent root reconstruction + granular state roots + body/header segregation #649)
  • Motivation
    • We want state transition function signature to look like (state, block) -> state but there was an implicit third parameter -- parent_root
    • This makes a number of small changes allowing us to cache the previous block header in state to make the state "self sufficient" in the state transition
    • To this end, we added latest_state_roots to state which is also a nice historic feature in general.
  • Changelog
    • replace Proposal with BeaconBlockHeader. Remove shard field.
      • note that hash_tree_root(block) == hash_tree_root(block_header)
    • change parent_root to previous_block_root in BeaconBlock
    • move eth1_data and randao_reveal into BeaconBlockBody
    • add latest_state_roots to BeaconState
    • change batched_block_roots to historical_roots and use this to batch (block, state) rather than just block. Do batching at end of epoch transition.
    • add get_temporary_block_header helper that produces a block header from a block with a ZERO_HASH for state root
    • modify get_genesis_beacon_state to handle modified state fields
    • per-slot processing
      • Do bookkeeping related to state, block, and latest_block_header
    • per-block processing
      • add "block header section" to handle all block header verification/bookkeeping
• get_previous_epoch returns current_epoch - 1 in all cases
• tie break for winning_root favors lexicographically higher hashes rather than lower
• Inactive slashed validators only pay inactivity penalties until withdrawable (ec12460) (bugfix)
• handle get_base_reward when no previous balance (02e8e89) (bugfix)
• handle case when no attestations available for a particular shard when crosslinking (47cca60) (bugfix)

[hwwhww]

Aw wow! I hope that will clarify a lot! 😊
Will take another look.

tip: "Add suggestion to batch" 😆

[CarlBeek]

So much more readable. 😍

[hwwhww]

and get_previous_epoch_attestations is undefined. 😅

[CarlBeek]

Standardises terminology across transfers:
to -> recipient

For some reason from was changed to sender for transfers earlier in this PR. I like this new terminology.

[protolambda]

There's currently 3 committee count functions:

get_previous_epoch_committee_count

def get_previous_epoch_committee_count(state: BeaconState) -> int:
    """
    Return the number of committees in the previous epoch of the given ``state``.
    """
    previous_active_validators = get_active_validator_indices(
        state.validator_registry,
        state.previous_shuffling_epoch,
    )
    return get_epoch_committee_count(len(previous_active_validators))

get_current_epoch_committee_count

def get_current_epoch_committee_count(state: BeaconState) -> int:
    """
    Return the number of committees in the current epoch of the given ``state``.
    """
    current_active_validators = get_active_validator_indices(
        state.validator_registry,
        state.current_shuffling_epoch,
    )
    return get_epoch_committee_count(len(current_active_validators))

get_next_epoch_committee_count

def get_next_epoch_committee_count(state: BeaconState) -> int:
    """
    Return the number of committees in the next epoch of the given ``state``.
    """
    next_active_validators = get_active_validator_indices(
        state.validator_registry,
        get_current_epoch(state) + 1,
    )
    return get_epoch_committee_count(len(next_active_validators))

Can we change it to just one function?

E.g.

def get_committee_count_for_epoch(state: BeaconState, epoch: Epoch) -> int:
    """
    Return the number of committees in the given ``epoch`` of the given ``state``.
    """
    active_validators = get_active_validator_indices(
        state.validator_registry,
        epoch,
    )
    return get_epoch_committee_count(len(active_validators))

And then usage is changed like:

get_previous_epoch_committee_count -> get_committee_count_for_epoch(state, state.previous_shuffling_epoch)
get_current_epoch_committee_count -> get_committee_count_for_epoch(state, state.current_shuffling_epoch)
get_next_epoch_committee_count ->  get_committee_count_for_epoch(state, get_current_epoch(state) + 1)

What do you think? Less functions is better?

Edit: we could also make len(active_validators) an argument, since the length may already be known before calling. Or reduce this all + get_epoch_committee_count to 1 function that has epoch and active_validator_count as arguments.

Edit 2: if we decide on len(active_validators) , we don't need the epoch argument.

[djrtwo]

@protolambda

There's currently 3 committee count functions:

Yeah, this is reasonable. I want to do any further changes outside of this PR. I'm working on a detailed changelog for implementers and it's already quite large :)

[protolambda]

FYI: I'm building an executable Go version of the spec, based on the spec from this PR, and really appreciate the new structuring. And yes, the PR is large, but it's worth it 100% 💯

[hwwhww]

Just some nitpicks.

Awesome work! 👍

[ralexstokes]

do we want to do some constant folding upon the final parameter tuning for these numbers? same below w/ INACTIVITY_PENALTY_QUOTIENT // 2 in get_inactivity_penalty

[salmanmahmud20]

Nice