Merge branch 'dev'

README.md

@@ -35,8 +35,7 @@ The current features are:
 
 ### Merge
 
-The merge is still actively in R&D. The specifications outline a general direction for engineering work,
-while the details are in review and may change.
+The merge is still actively in development. The exact specification has not been formally accepted as final and details are still subject to change.
 
 * Background material:
   * An [ethresear.ch](https://ethresear.ch) post [describing the basic mechanism](https://ethresear.ch/t/the-eth1-eth2-transition/6265)

setup.py

@@ -223,7 +223,7 @@ def get_spec(file_name: Path, preset: Dict[str, str], config: Dict[str, str]) ->
 
                     if not _is_constant_id(name):
                         # Check for short type declarations
-                        if value.startswith("uint") or value.startswith("Bytes") or value.startswith("ByteList"):
+                        if value.startswith("uint") or value.startswith("Bytes") or value.startswith("ByteList") or value.startswith("Union"):
                             custom_types[name] = value
                         continue
 
@@ -495,7 +495,7 @@ def imports(cls, preset_name: str):
         return super().imports(preset_name) + f'''
 from typing import Protocol
 from eth2spec.phase0 import {preset_name} as phase0
-from eth2spec.utils.ssz.ssz_typing import Bytes20, ByteList, ByteVector, uint256
+from eth2spec.utils.ssz.ssz_typing import Bytes20, ByteList, ByteVector, uint256, Union
 '''
 
     @classmethod
@@ -523,7 +523,7 @@ def get_pow_chain_head() -> PowBlock:
 
 class NoopExecutionEngine(ExecutionEngine):
 
-    def new_block(self, execution_payload: ExecutionPayload) -> bool:
+    def on_payload(self, execution_payload: ExecutionPayload) -> bool:
         return True
 
     def set_head(self, block_hash: Hash32) -> bool:
@@ -532,7 +532,7 @@ def set_head(self, block_hash: Hash32) -> bool:
     def finalize_block(self, block_hash: Hash32) -> bool:
         return True
 
-    def assemble_block(self, block_hash: Hash32, timestamp: uint64) -> ExecutionPayload:
+    def assemble_block(self, block_hash: Hash32, timestamp: uint64, random: Bytes32) -> ExecutionPayload:
         raise NotImplementedError("no default block production")
 
 
@@ -553,6 +553,10 @@ def hardcoded_custom_type_dep_constants(cls) -> str:
 }
 
 
+def is_spec_defined_type(value: str) -> bool:
+    return value.startswith('ByteList') or value.startswith('Union')
+
+
 def objects_to_spec(preset_name: str,
                     spec_object: SpecObject,
                     builder: SpecBuilder,
@@ -565,15 +569,15 @@ def objects_to_spec(preset_name: str,
             [
                 f"class {key}({value}):\n    pass\n"
                 for key, value in spec_object.custom_types.items()
-                if not value.startswith('ByteList')
+                if not is_spec_defined_type(value)
             ]
         )
-        + ('\n\n' if len([key for key, value in spec_object.custom_types.items() if value.startswith('ByteList')]) > 0 else '')
+        + ('\n\n' if len([key for key, value in spec_object.custom_types.items() if is_spec_defined_type(value)]) > 0 else '')
         + '\n\n'.join(
             [
                 f"{key} = {value}\n"
                 for key, value in spec_object.custom_types.items()
-                if value.startswith('ByteList')
+                if is_spec_defined_type(value)
             ]
         )
     )
@@ -1020,7 +1024,7 @@ def run(self):
         "py_ecc==5.2.0",
         "milagro_bls_binding==1.6.3",
         "dataclasses==0.6",
-        "remerkleable==0.1.20",
+        "remerkleable==0.1.21",
         RUAMEL_YAML_VERSION,
         "lru-dict==1.1.6",
         MARKO_VERSION,

specs/altair/p2p-interface.md

@@ -137,13 +137,14 @@ def get_sync_subcommittee_pubkeys(state: BeaconState, subcommittee_index: uint64
     return sync_committee.pubkeys[i:i + sync_subcommittee_size]
 ```
 
-- _[IGNORE]_ The contribution's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `contribution.slot == current_slot`.
-- _[IGNORE]_ The block being signed over (`contribution.beacon_block_root`) has been seen (via both gossip and non-gossip sources).
+- _[IGNORE]_ The contribution's slot is for the current slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance), i.e. `contribution.slot == current_slot`.
 - _[REJECT]_ The subcommittee index is in the allowed range, i.e. `contribution.subcommittee_index < SYNC_COMMITTEE_SUBNET_COUNT`.
-- _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index` for the slot `contribution.slot` and subcommittee index `contribution.subcommittee_index`.
 - _[REJECT]_ `contribution_and_proof.selection_proof` selects the validator as an aggregator for the slot -- i.e. `is_sync_committee_aggregator(contribution_and_proof.selection_proof)` returns `True`.
 - _[REJECT]_ The aggregator's validator index is in the declared subcommittee of the current sync committee --
   i.e. `state.validators[contribution_and_proof.aggregator_index].pubkey in get_sync_subcommittee_pubkeys(state, contribution.subcommittee_index)`.
+- _[IGNORE]_ The sync committee contribution is the first valid contribution received for the aggregator with index `contribution_and_proof.aggregator_index`
+  for the slot `contribution.slot` and subcommittee index `contribution.subcommittee_index`
+  (this requires maintaining a cache of size `SYNC_COMMITTEE_SIZE` for this topic that can be flushed after each slot).
 - _[REJECT]_ The `contribution_and_proof.selection_proof` is a valid signature of the `SyncAggregatorSelectionData` derived from the `contribution` by the validator with index `contribution_and_proof.aggregator_index`.
 - _[REJECT]_ The aggregator signature, `signed_contribution_and_proof.signature`, is valid.
 - _[REJECT]_ The aggregate signature is valid for the message `beacon_block_root` and aggregate pubkey derived from the participation info in `aggregation_bits` for the subcommittee specified by the `contribution.subcommittee_index`.
@@ -158,12 +159,12 @@ The `sync_committee_{subnet_id}` topics are used to propagate unaggregated sync
 
 The following validations MUST pass before forwarding the `sync_committee_message` on the network:
 
-- _[IGNORE]_ The signature's slot is for the current slot (with a MAXIMUM_GOSSIP_CLOCK_DISPARITY allowance), i.e. `sync_committee_message.slot == current_slot`.
-- _[IGNORE]_ The block being signed over (`sync_committee_message.beacon_block_root`) has been seen (via both gossip and non-gossip sources).
-- _[IGNORE]_ There has been no other valid sync committee signature for the declared `slot` for the validator referenced by `sync_committee_message.validator_index`.
-  Note this validation is _per topic_ so that for a given `slot`, multiple messages could be forwarded with the same `validator_index` as long as the `subnet_id`s are distinct.
+- _[IGNORE]_ The signature's slot is for the current slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance), i.e. `sync_committee_message.slot == current_slot`.
 - _[REJECT]_ The `subnet_id` is valid for the given validator, i.e. `subnet_id in compute_subnets_for_sync_committee(state, sync_committee_message.validator_index)`.
   Note this validation implies the validator is part of the broader current sync committee along with the correct subcommittee.
+- _[IGNORE]_ There has been no other valid sync committee signature for the declared `slot` for the validator referenced by `sync_committee_message.validator_index`
+  (this requires maintaining a cache of size `SYNC_COMMITTEE_SIZE // SYNC_COMMITTEE_SUBNET_COUNT` for each subnet that can be flushed after each slot).
+  Note this validation is _per topic_ so that for a given `slot`, multiple messages could be forwarded with the same `validator_index` as long as the `subnet_id`s are distinct.
 - _[REJECT]_ The `signature` is valid for the message `beacon_block_root` for the validator referenced by `validator_index`.
 
 #### Sync committees and aggregation

specs/altair/validator.md

@@ -265,7 +265,7 @@ This process occurs each slot.
 
 ##### Prepare sync committee message
 
-If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every `slot` in the current sync committee period, the validator should prepare a `SyncCommitteeMessage` for the previous slot (`slot - 1`) according to the logic in `get_sync_committee_message` as soon as they have determined the head block of `slot - 1`.
+If a validator is in the current sync committee (i.e. `is_assigned_to_sync_committee()` above returns `True`), then for every `slot` in the current sync committee period, the validator should prepare a `SyncCommitteeMessage` for the previous slot (`slot - 1`) according to the logic in `get_sync_committee_message` as soon as they have determined the head block of `slot - 1`. This means that when assigned to `slot` a `SyncCommitteeMessage` is prepared and broadcast in `slot-1 ` instead of `slot`.   
 
 This logic is triggered upon the same conditions as when producing an attestation.
 Meaning, a sync committee member should produce and broadcast a `SyncCommitteeMessage` either when (a) the validator has received a valid block from the expected block proposer for the current `slot` or (b) one-third of the slot has transpired (`SECONDS_PER_SLOT / 3` seconds after the start of the slot) -- whichever comes first.