Kintsugi 🍵 #2768

paulhauner · 2021-11-03T04:56:49Z

Description

This is the canonical branch for the Kintsugi testnet.

For instructions on running the Kintsugi testnet, see #2830.

This PR contains the changes required for Lighthouse to run "the merge" specifications. It does not deliver a production-ready implementation of "the merge" for Lighthouse, instead it provides the necessary components for Lighthouse to run on testnets and progress towards a production implementation.

Where this PR touches code that will run on the pre-merge (i.e., current) Beacon Chain, any changes should be production ready. For post-merge components (e.g., execution_layer), some degree of unfinished-ness is acceptable. After all, "the merge" specification is not yet complete.

TODO

This section lists tasks required before this branch can be merged into unstable.

N/A

realbigsean · 2021-11-04T14:59:56Z

Upgrade to v1.1.4 spec tests, fixing any consensus changes.

I can work on this

paulhauner · 2021-11-11T05:21:00Z

I just rebased this branch onto unstable. It will fail CI until I push another fix (#2799).

beacon_node/execution_layer/src/engine_api/http.rs

paulhauner · 2021-11-28T22:53:07Z

I just rebased to fix a conflict with unstable and to fix a merge (instead of squash-merge) of #2796.

This will fail CI until we merge #2829.

michaelsproul

I have reviewed the ./consensus changes thoroughly and I'm happy to merge this to unstable.

Makefile

beacon_node/beacon_chain/src/beacon_chain.rs

consensus/proto_array/src/proto_array_fork_choice.rs

consensus/proto_array/src/ssz_container.rs

consensus/types/Cargo.toml

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs

beacon_node/src/config.rs

divagant-martian

I checked ./lighthouse_network and ./network. it looks good to me

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs

beacon_node/lighthouse_network/src/rpc/protocol.rs

common/slot_clock/src/lib.rs

Added Execution Payload from Rayonism Fork Updated new Containers to match Merge Spec Updated BeaconBlockBody for Merge Spec Completed updating BeaconState and BeaconBlockBody Modified ExecutionPayload<T> to use Transaction<T> Mostly Finished Changes for beacon-chain.md Added some things for fork-choice.md Update to match new fork-choice.md/fork.md changes ran cargo fmt Added Missing Pieces in eth2_libp2p for Merge fix ef test Various Changes to Conform Closer to Merge Spec

* Update to v1.1.0-beta.4 (squash of #2548) * SSZ, cached tree hash, EF tests

* Remove unchecked arith from ssz_derive * Address clippy lints in block_verfication * Use safe math for is_valid_gas_limit

* Add payload verification status to fork choice * Pass payload verification status to import_block * Add valid back-propagation * Add head safety status latch to API * Remove ExecutionLayerStatus * Add execution info to client notifier * Update notifier logs * Change use of "hash" to refer to beacon block * Shutdown on invalid finalized block * Tidy, add comments * Fix failing FC tests * Allow blocks with unsafe head * Fix forkchoiceUpdate call on startup

* update initializing from eth1 for merge genesis * read execution payload header from file lcli * add `create-payload-header` command to `lcli` * fix base fee parsing * Apply suggestions from code review * default `execution_payload_header` bool to false when deserializing `meta.yml` in EF tests Co-authored-by: Paul Hauner <paul@paulhauner.com>

* Freshen Cargo.lock * Fix gossip worker * Update map_fork_name_with

* Fix arbitrary check kintsugi * Add merge chain spec fields, and a function to determine which constant to use based on the state variant * increment spec test version * Remove `Transaction` enum wrapper * Remove Transaction new-type * Remove gas validations * Add `--terminal-block-hash-epoch-override` flag * Increment spec tests version to 1.1.5 * Remove extraneous gossip verification ethereum/consensus-specs#2687 * - Remove unused Error variants - Require both "terminal-block-hash-epoch-override" and "terminal-block-hash-override" when either flag is used * - Remove a couple more unused Error variants Co-authored-by: Paul Hauner <paul@paulhauner.com>

* Change base_fee_per_gas to Uint256 * Add custom (de)serialization to ExecutionPayload * Fix errors * Add a quoted_u256 module * Remove unused function * lint * Add test * Remove extra line Co-authored-by: Paul Hauner <paul@paulhauner.com>

* Fix fork choice after rebase * Remove paulhauner warp dep * Fix fork choice test compile errors * Assume fork choice payloads are valid * Add comment * Ignore new tests * Fix error in test skipping

* Allocate less memory (3GB) in engine tests * Run cargo format * Remove tx too large test Co-authored-by: Michael Sproul <michael@sigmaprime.io>

* Added ForkchoiceUpdatedV1 & GetPayloadV1 * Added ExecutePayloadV1 * Added new geth test vectors * Separated Json Object/Serialization Code into file * Deleted code/tests for Requests Removed from spec * Finally fixed serialization of null '0x' * Made Naming of JSON Structs Consistent * Fix clippy lints * Remove u64 payload id * Remove unused serde impls * Swap to [u8; 8] for payload id * Tidy * Adjust some block gen return vals * Tidy * Add fallback when payload id is unknown * Remove comment Co-authored-by: Mark Mackey <mark@sigmaprime.io>

* Unify loading of eth2_network_config * Apply overrides at lighthouse binary level * Remove duplicate override values * Add merge values to existing net configs * Make override flags global * Add merge fields to testing config * Add one to TTD * Fix failing engine tests * Fix test compile error * Remove TTD flags * Move get_eth2_network_config * Fix warn * Address review comments

Fix max packet sizes Fix max_payload_size function Add merge block test Fix max size calculation; fix up test Clear comments Add a payload_size_function Use safe arith for payload calculation Return an error if block too big in block production Separate test to check if block is over limit

* Start v1.1.5 updates * Implement new payload creation logic * Tidy, add comments * Remove unused error enums * Add validate payload for gossip * Refactor validate_merge_block * Split payload verification in per block processing * Add execute_payload * Tidy * Tidy * Start working on new fork choice tests * Fix failing merge block test * Skip block_lookup_failed test * Fix failing terminal block test * Fixes from self-review * Address review comments

* Fix makefile * Return on invalid finalized block * Fix todo in gossip scoring * Require --merge for --fee-recipient * Bump eth2_serde_utils * Change schema versions * Swap hash/uint256 test_random impls * Use default for ExecutionPayload::empty * Check for DBs before removing * Remove kintsugi docker image * Fix CLI default value

* Remove TODOs * Fix typo

paulhauner · 2021-12-02T03:32:45Z

I've just rebased to fix a merge conflict.

paulhauner added the kintsugi 🍵 label Nov 3, 2021

paulhauner changed the base branch from stable to unstable November 3, 2021 04:57

paulhauner mentioned this pull request Nov 3, 2021

Merge F2F #2620

Closed

7 tasks

paulhauner force-pushed the kintsugi branch from 06bcbdd to 4b0ab58 Compare November 3, 2021 05:23

paulhauner force-pushed the kintsugi branch from f817784 to 954a9c2 Compare November 11, 2021 05:20

michaelsproul reviewed Nov 11, 2021

View reviewed changes

beacon_node/execution_layer/src/engine_api/http.rs Outdated Show resolved Hide resolved

michaelsproul mentioned this pull request Nov 11, 2021

Allocate less memory (3GB) in engine tests #2800

Closed

paulhauner mentioned this pull request Nov 15, 2021

Skip memory intensive engine test #2809

Merged

paulhauner force-pushed the kintsugi branch 2 times, most recently from 553ea8b to 6f2a3c6 Compare November 28, 2021 22:53

michaelsproul added the backwards-incompat Backwards-incompatible API change label Nov 28, 2021

paulhauner mentioned this pull request Nov 28, 2021

Kintsugi Tracking 🍵 #2830

Closed

12 tasks

michaelsproul approved these changes Nov 28, 2021

View reviewed changes

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs Outdated Show resolved Hide resolved

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/lighthouse_network/src/rpc/codec/ssz_snappy.rs Outdated Show resolved Hide resolved

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/src/config.rs Outdated Show resolved Hide resolved

paulhauner commented Nov 29, 2021

View reviewed changes

beacon_node/src/config.rs Outdated Show resolved Hide resolved

divagant-martian approved these changes Nov 29, 2021

View reviewed changes

paulhauner mentioned this pull request Nov 30, 2021

Kintsugi Diva comments #2836

Merged

michaelsproul reviewed Nov 30, 2021

View reviewed changes

common/slot_clock/src/lib.rs Outdated Show resolved Hide resolved

This was referenced Nov 30, 2021

[Merged by Bors] - Retrospective invalidation of exec. payloads for opt. sync #2837

Closed

Kintsugi slot clock fixes #2842

Merged

ethDreamer and others added 5 commits December 2, 2021 14:26

Fix consensus, SSZ, tree hash & run merge EF tests (#2622)

cce855f

* Update to v1.1.0-beta.4 (squash of #2548) * SSZ, cached tree hash, EF tests

Fix clippy lints on merge-f2f (#2626)

c10e8ce

* Remove unchecked arith from ssz_derive * Address clippy lints in block_verfication * Use safe math for is_valid_gas_limit

Update merge consensus to v1.1.0-beta.5 (#2630)

10b263f

Handle merge fork in web3signer (#2631)

18eee2d

pawanjay176 and others added 24 commits December 2, 2021 14:26

Fix db paths when datadir is relative (#2682)

aa1d57a

Accept TTD override as decimal (#2676)

6b4cc63

Kintsugi rebase patches (#2769)

86e0c56

* Freshen Cargo.lock * Fix gossip worker * Update map_fork_name_with

Move merge-f2f docker to kintsugi (#2774)

cee18ca

Fix arbitrary check kintsugi (#2777)

cdbe603

Fix Uint256 deserialization (#2786)

24966c0

* Change base_fee_per_gas to Uint256 * Add custom (de)serialization to ExecutionPayload * Fix errors * Add a quoted_u256 module * Remove unused function * lint * Add test * Remove extra line Co-authored-by: Paul Hauner <paul@paulhauner.com>

Remove old uses of testnet

eb35c64

Fixes after rebasing Kintsugi onto unstable (#2799)

cbd2201

* Fix fork choice after rebase * Remove paulhauner warp dep * Fix fork choice test compile errors * Assume fork choice payloads are valid * Add comment * Ignore new tests * Fix error in test skipping

Skip memory intensive engine test (#2809)

cdfd130

* Allocate less memory (3GB) in engine tests * Run cargo format * Remove tx too large test Co-authored-by: Michael Sproul <michael@sigmaprime.io>

Removed PowBlock struct that never got used (#2813)

f674853

Bump crate versions (#2829)

82a8152

Cleanup Comments & Fix get_pow_block_hash_at_ttd() (#2835)

c2f2813

Kintsugi Diva comments (#2836)

ab86b42

* Remove TODOs * Fix typo

Support legacy data directories (#2846)

94385fe

Remove duplicate slot_clock method (#2842)

144978f

Restrict network limits based on merge fork epoch (#2839)

f3c237c

paulhauner force-pushed the kintsugi branch from 2d8650b to f3c237c Compare December 2, 2021 03:32

paulhauner merged commit f3c237c into unstable Dec 2, 2021

paulhauner deleted the kintsugi branch December 2, 2021 05:51

paulhauner restored the kintsugi branch December 2, 2021 05:52

AgeManning deleted the kintsugi branch January 20, 2022 21:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kintsugi 🍵 #2768

Kintsugi 🍵 #2768

paulhauner commented Nov 3, 2021 •

edited

Loading

realbigsean commented Nov 4, 2021

paulhauner commented Nov 11, 2021 •

edited

Loading

paulhauner commented Nov 28, 2021 •

edited

Loading

michaelsproul left a comment

divagant-martian left a comment

paulhauner commented Dec 2, 2021

Kintsugi 🍵 #2768

Kintsugi 🍵 #2768

Conversation

paulhauner commented Nov 3, 2021 • edited Loading

Description

TODO

realbigsean commented Nov 4, 2021

paulhauner commented Nov 11, 2021 • edited Loading

paulhauner commented Nov 28, 2021 • edited Loading

michaelsproul left a comment

Choose a reason for hiding this comment

divagant-martian left a comment

Choose a reason for hiding this comment

paulhauner commented Dec 2, 2021

paulhauner commented Nov 3, 2021 •

edited

Loading

paulhauner commented Nov 11, 2021 •

edited

Loading

paulhauner commented Nov 28, 2021 •

edited

Loading