Compact committee roots #1219

JustinDrake · 2019-06-26T12:23:02Z

Replacement of active_index_roots with compact_committee_roots to allow for significantly more efficient light clients (see item 17 in #1054). Key changes:

Validator information (including indices) is pre-shuffled and grouped by crosslink committee.
Validator information relevant to light clients (validator.effective_balance and validator.slashed) is kept "close" to the validator index to avoid accessing state.validators.
Validator information (index, effective_balance, slashed) is compactified into a uint64 which is friendly to SSZ packing.
Shuffled pubkeys are available but kept separate from compact validator information. Light clients that have pubkeys cached do not need to re-download and re-authenticate them.

The plan is to also have persistent committee roots in phase 1.

djrtwo

cool!

specs/core/0_beacon-chain.md

JustinDrake · 2019-06-27T10:08:08Z

@protolambda Testing goes a little crazy on this branch. Not sure what to do.

protolambda · 2019-06-27T13:56:41Z

I'm seeing the same on this branch, here's my test output:
https://gist.github.com/protolambda/f60b5eecba89a4f29367ae9afc76c725

djrtwo · 2019-06-27T16:06:50Z

Python crashes are often recursion bugs

djrtwo · 2019-06-27T16:36:20Z

taking a look at this

djrtwo · 2019-06-27T16:43:21Z

You have a circular dependency

get_compact_committees_root -> get_crosslink_committee -> generate_seed -> get_compact_committees_root

Suggestion:

Use active_index_root in generate_seed instead of the committee root

def generate_seed(state: BeaconState,
                  epoch: Epoch) -> Hash:
    """
    Generate a seed for the given ``epoch``.
    """
    return hash(
        get_randao_mix(state, Epoch(epoch + EPOCHS_PER_HISTORICAL_VECTOR - MIN_SEED_LOOKAHEAD)) +
        hash_tree_root(get_active_validator_indices(state, epoch)) +
        int_to_bytes(epoch, length=32)
    )

djrtwo · 2019-06-27T22:35:41Z

I fixed the SSZ typing error, but there is a more substantive error now. We are trying to set compact_committees for an epoch ACTIVATION_EXIT_EPOCHs in the future. We can;t actually reliably do this because we have MIN_SEED_LOOKAHEAD.

It seems more appropriate to just set for MIN_SEED_LOOKAHEAD epochs in the future because that is the most immediate epoch available that we know both the v-set and the seed (requirements for computing the committees).

Setting it to this value actually triggers an issue in get_epoch_start_shard which needs to be adjusted to work for epochs more than just the next. Need to think this over a bit.

JustinDrake · 2019-06-28T08:44:12Z

motes.md

@@ -0,0 +1,42 @@
+* `BLS_WITHDRAWAL_PREFIX`


Loving your motes :) Will comment on the various points later today 😂

Oh no! I accidentally pushed. Haaa

JustinDrake · 2019-06-28T09:56:29Z

motes.md

@@ -0,0 +1,42 @@
+* `BLS_WITHDRAWAL_PREFIX`
+  * Why int rather than bytes?


Should probably be bytes :)

JustinDrake · 2019-06-28T09:57:23Z

motes.md

+* `BLS_WITHDRAWAL_PREFIX`
+  * Why int rather than bytes?
+* `MIN_SEED_LOOKAHEAD`
+  * Is this actually tunable?


I don't any reason why it's not tunable

JustinDrake · 2019-06-28T09:58:39Z

motes.md

+  * Why int rather than bytes?
+* `MIN_SEED_LOOKAHEAD`
+  * Is this actually tunable?
+  * If so, what are the reprecussions?


I can think of two repercussions:

It changes the lookahead (duh)

For security we need to correspondingly adjust ACTIVATION_EXIT_DELAY to be at least MIN_SEED_LOOKAHEAD + 3.

JustinDrake · 2019-06-28T09:59:50Z

motes.md

+* `ACTIVATION_EXIT_DELAY`
+  * Reaquaint with purpose
+* AttesterSlashings
+  * `MAX_ATTESTER_SLASHINGS` is 1.


Right, the reasons being:

1 is sufficiently large

Attester slashings are huge objects, and could be used as a DoS vector

JustinDrake · 2019-06-28T10:00:47Z

motes.md

+  * Are there scenarios in which validators can create more effective slashable
+    messages than can be included on chain? For example, Validators split up to
+    create double attestations for checkpoints but different (junk) crosslink
+    data to prevent them from being aggregatable to the fullest


I remember thinking about this. One way to create more effective slashable messages is as follows. Let's say you control n validators, and you want to make bad attestations a_1, ..., a_n for each of these validators. The attacker can broadcast aggregated attestations (a_1, a_i) for i > 1 and never reveal the individual a_i. This is an argument for generalising the bitfield to a multibitfield.

JustinDrake · 2019-06-28T10:13:54Z

motes.md

+  * checking next hinges upon the fact that the validator set for the next
+    epoch is 100% known at the current epoch. Ensure this is the case
+* `get_block_root_at_slot` .. `generate_seed` can be bade into one line
+  function signatures


Yay for one-line function signatures. This is part of the cleanup planned in #918.

JustinDrake · 2019-06-28T10:14:43Z

motes.md

+  * is the `2**40` special for security of alg? probably.
+
+
+pubkey/privkey g1 vs g2


pubkey should be in g1 since we're adding a bunch of them when aggregating.

JustinDrake · 2019-06-28T10:18:06Z

motes.md

+* `get_block_root_at_slot` .. `generate_seed` can be bade into one line
+  function signatures
+* `get_shuffled_index`
+  * I think it should be maybe `assert index_count <= VALIDATOR_REGISTRY_LIMIT`


JustinDrake · 2019-06-28T10:18:19Z

motes.md

+  function signatures
+* `get_shuffled_index`
+  * I think it should be maybe `assert index_count <= VALIDATOR_REGISTRY_LIMIT`
+  * is the `2**40` special for security of alg? probably.


Hum, not sure.

JustinDrake · 2019-06-28T10:20:30Z

motes.md

+  * Is this actually tunable?
+  * If so, what are the reprecussions?
+* `ACTIVATION_EXIT_DELAY`
+  * Reaquaint with purpose


The problem is that attackers can know the randomness beyond MIN_SEED_LOOKAHEAD, and hence bias the shuffling by strategically registering/deregistering validators in the window between MIN_SEED_LOOKAHEAD and MIN_SEED_LOOKAHEAD + 3 to "grind" the shuffling. (The number 3 is large enough to guarantee that, with high probability, an attacker's lookahead cannot go beyond MIN_SEED_LOOKAHEAD + 3.)

…in the future

CarlBeek

LGTM. 👍

notes.txt and motes.md should be removed before merge though.

notes.txt

motes.md

specs/core/0_beacon-chain.md

ralexstokes

i'm excited to see how this broadens the light client design space :)

found a typo

specs/light_client/sync_protocol.md

Co-Authored-By: Alex Stokes <r.alex.stokes@gmail.com>

specs/core/0_beacon-chain.md

@NIC619

Thanks @NIC619 #1219 (comment)

Initial draft

8c34aa8

JustinDrake mentioned this pull request Jun 26, 2019

Active registry roots #1104

Closed

JustinDrake added the milestone:June 30 freeze 🥶 Phase 0 spec freeze for long-lived cross-client testnet label Jun 26, 2019

djrtwo reviewed Jun 27, 2019

View reviewed changes

specs/core/0_beacon-chain.md Outdated Show resolved Hide resolved

specs/core/0_beacon-chain.md Outdated Show resolved Hide resolved

specs/core/0_beacon-chain.md Outdated Show resolved Hide resolved

WIP

bcfe383

JustinDrake and others added 2 commits June 27, 2019 19:05

Apply Danny's suggestions

853f5fc

fix committee typing error

990cc55

JustinDrake commented Jun 28, 2019

View reviewed changes

djrtwo added 3 commits June 28, 2019 11:10

set committees root for next epoch rather tahn ACTIVaTION_EXIT_DELAY …

5a8f3e4

…in the future

use active index root for generate seed mix in

b40e228

lint

9993a28

djrtwo marked this pull request as ready for review June 28, 2019 17:31

Merge branch 'dev' into committee-roots

b88ab25

CarlBeek approved these changes Jun 28, 2019

View reviewed changes

notes.txt Outdated Show resolved Hide resolved

motes.md Outdated Show resolved Hide resolved

specs/core/0_beacon-chain.md Show resolved Hide resolved

remove local notes files

d5c2ecb

ralexstokes approved these changes Jun 29, 2019

View reviewed changes

specs/light_client/sync_protocol.md Outdated Show resolved Hide resolved

djrtwo and others added 2 commits June 29, 2019 12:04

minor typo

bc8df3c

Co-Authored-By: Alex Stokes <r.alex.stokes@gmail.com>

Merge branch 'dev' into committee-roots

2857324

djrtwo merged commit e1826aa into dev Jun 29, 2019

djrtwo deleted the committee-roots branch June 29, 2019 17:26

NIC619 reviewed Jun 30, 2019

View reviewed changes

specs/core/0_beacon-chain.md Show resolved Hide resolved

JustinDrake added a commit that referenced this pull request Jun 30, 2019

Fix typo

6cfd3b5

Thanks @NIC619 #1219 (comment)

JustinDrake mentioned this pull request Jun 30, 2019

Fix typo #1234

Merged

terencechain mentioned this pull request Jul 1, 2019

[Mega Tracking] - Align to spec freeze prysmaticlabs/prysm#2891

Closed

19 tasks

nisdas mentioned this pull request Jul 2, 2019

Implement Compact Committee Root prysmaticlabs/prysm#2897

Merged

3 tasks

hwwhww mentioned this pull request Jul 5, 2019

Updates for almost v0.8 ethereum/trinity#714

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compact committee roots #1219

Compact committee roots #1219

JustinDrake commented Jun 26, 2019

djrtwo left a comment

JustinDrake commented Jun 27, 2019

protolambda commented Jun 27, 2019

djrtwo commented Jun 27, 2019

djrtwo commented Jun 27, 2019

djrtwo commented Jun 27, 2019 •

edited

Loading

djrtwo commented Jun 27, 2019

JustinDrake Jun 28, 2019

djrtwo Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

JustinDrake Jun 28, 2019

CarlBeek left a comment

ralexstokes left a comment

		@@ -0,0 +1,42 @@
		* `BLS_WITHDRAWAL_PREFIX`
		* Why int rather than bytes?

		* is the `2**40` special for security of alg? probably.


		pubkey/privkey g1 vs g2

Compact committee roots #1219

Compact committee roots #1219

Conversation

JustinDrake commented Jun 26, 2019

djrtwo left a comment

Choose a reason for hiding this comment

JustinDrake commented Jun 27, 2019

protolambda commented Jun 27, 2019

djrtwo commented Jun 27, 2019

djrtwo commented Jun 27, 2019

djrtwo commented Jun 27, 2019 • edited Loading

djrtwo commented Jun 27, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CarlBeek left a comment

Choose a reason for hiding this comment

ralexstokes left a comment

Choose a reason for hiding this comment

djrtwo commented Jun 27, 2019 •

edited

Loading