Remove shards and committees #3896

terencechain · 2019-10-30T05:48:38Z

Part of #3865

This PR removes shards and crosslinks. Excuse the big diff, they are all due to the deletion of pbs, crosslink spec tests and all the crosslink transition function/test.

Change list:

Modified CommitteeCount to CommitteeCountAtSlot
Renamed crosslink committee to beacon committee
Removed AttestationDataSlot
Removed ProcessCrosslinks
Removed WinningCrosslink
Removed CrosslinkDelta
Removed ShardDelta
Removed StartShard
Removed attsForCrosslink
Removed crosslinkDeltaPreCompute
Removed crosslink in state
Removed crosslink spec tests
Updated block transition ProcessAttestation to neglect crosslink
Updated state transition to neglect crosslink
Updated configs (12s in a slot, 32 slots per epoch, max validators per committee is 2048)

…ots-removal

codecov · 2019-10-30T21:31:01Z

Codecov Report

Merging #3896 into v0.9 will decrease coverage by 24.67%.
The diff coverage is 74.24%.

@@            Coverage Diff             @@
##            v0.9    #3896       +/-   ##
==========================================
- Coverage   46.8%   22.12%   -24.68%     
==========================================
  Files        157      134       -23     
  Lines      10022     8039     -1983     
==========================================
- Hits        4691     1779     -2912     
- Misses      4658     5989     +1331     
+ Partials     673      271      -402

prestonvanloon · 2019-10-30T22:53:16Z

shared/params/config.go

 	TargetCommitteeSize:            128,
-	MaxValidatorsPerCommittee:      4096,
+	MaxValidatorsPerCommittee:      2048,


Why is this different? This isn't mentioned in your description.

Updated descriptions

prestonvanloon · 2019-10-30T22:53:23Z

shared/params/config.go

@@ -134,8 +134,8 @@ var defaultBeaconConfig = &BeaconChainConfig{

 	// Time parameter constants.
 	MinAttestationInclusionDelay:     1,
-	SecondsPerSlot:                   6,
-	SlotsPerEpoch:                    64,
+	SecondsPerSlot:                   12,


Updated descriptions, the rationale is more events will occur per slot in these modified proposals so upped 6 to 12

prestonvanloon · 2019-10-30T22:56:04Z

beacon-chain/sync/validate_beacon_blocks_test.go

@@ -98,7 +98,6 @@ func TestValidateBeaconBlockPubSub_BlockAlreadyPresentInCache(t *testing.T) {
 		t.Fatal(err)
 	}
 	msg := &ethpb.BeaconBlock{
-		Slot:       1,


Why these changes in beacon-chain/sync/validate_beacon_blocks_test.go?

With slot 1, we get an error could not process slot from the future, slot time 1572483483 > current time 1572483471.
Which makes sense because genesis_time is time.Now and we are passing in a block with slot 1 right away. This doesn't happen with 6s slot but only with 12s slot

prestonvanloon · 2019-10-30T22:57:52Z

beacon-chain/sync/initial-sync/round_robin_test.go

@@ -87,29 +87,29 @@ func TestRoundRobinSync(t *testing.T) {
 		},
 		{
 			name:               "Multiple peers with failures",
-			currentSlot:        320, // 5 epochs
-			expectedBlockSlots: makeSequence(1, 320),
+			currentSlot:        160, // 10 epochs


Can you leave the number of slots the same, but just change the epochs? It looks like you reduced the blocks by half and also the finalized epoch by half in the peer data

Np. Since epoch duration is 32 slots. I'm going to use 320 blocks and just increased epoch to 8 (double from before)

prestonvanloon · 2019-10-30T22:59:43Z

beacon-chain/core/state/state_test.go

@@ -15,7 +15,7 @@ import (
 )

 func TestGenesisBeaconState_OK(t *testing.T) {
-	if params.BeaconConfig().SlotsPerEpoch != 64 {
+	if params.BeaconConfig().SlotsPerEpoch != 32 {
 		t.Errorf("SlotsPerEpoch should be 64 for these tests to pass")


Suggested change

t.Errorf("SlotsPerEpoch should be 64 for these tests to pass")

t.Errorf("SlotsPerEpoch should be 32 for these tests to pass")

I think you can just delete this assertion. Seems like it had something to do with shard count

prestonvanloon · 2019-10-30T23:00:21Z

beacon-chain/core/helpers/validators_test.go

@@ -144,7 +144,7 @@ func TestIsSlashableValidator_InactiveSlashed(t *testing.T) {
 func TestBeaconProposerIndex_OK(t *testing.T) {
 	ClearAllCaches()

-	if params.BeaconConfig().SlotsPerEpoch != 64 {
+	if params.BeaconConfig().SlotsPerEpoch != 32 {
 		t.Errorf("SlotsPerEpoch should be 64 for these tests to pass")


Delete these assertions if you could change this number and no tests fail. We had this because we wanted to fail faster on config changes

* Update seed domains (#3872) * Remove Transfers (#3870) * Remove active index roots and compact committee roots (#3869) * Update inclusion reward (#3886) * Alter proposer selection logic (#3884) * Fix early committee bias (#3888) * Remove shards and committees (#3896) * Epoch spec tests v0.9 (#3907) * Block spec test v0.9 (#3905) * rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes * Update deposit contract (#3906) * Proto spec tests v0.9 (#3908) * Starting * Add AggregateAndProof * Unskip block util tests (#3910) * rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes * Unskip block util tests * Slot processing spec test V0.9 (#3912) * Starting * Add AggregateAndProof * Unskip slot processing mainnet test * Unskip minimal spec test for finalization (#3920) * Remove outdated interop tests (#3922) * Rm outdated interop tests * Rm test runner * Gazelle * Update validator to use proposer slot (#3919) * Fix committee assignment (#3931) * Replace shard with committee index (#3930) * Conflict * Clean up (#3933) * Remove shard filter in db (#3936) * Remove lightouse compatibility test (#3939) * Update Committee Cache for v0.9 (#3948) * Updated committee cache * Removed shuffled indices cache * Started testing run time * Lint * Fixed test * Safeguard against nil head state * address edge case * add test * Fixed TestRoundRobinSync by doubling the epochs * Unskip TestProtoCompatability (#3958) * Unskip TestProtoCompatability * Update WORKSPACE * Fix minimal config (#3959) * fix minimal configs * fix hardcoded value in test * Simplify verify att time (#3961) * update readme for deposit contract, regen bindings for vyper 0.1.0b12 (#3963) * update readme for deposit contract, regen bindings * medium * Check nil base state (#3964) * Copy Block When Receiving it From Sync (#3966) * copy block * clone for other service methods too * Change logging of Bitfield (#3956) * change logging of bits * preston's review * Unskip Beacon Server Test (#3962) * run test till the end * fix up proto message types * fmt * resolve broken tests * better error handling * fixing new logic to use archived proposer info * fix up logic * clip using the max effective balance * broken build fix with num arg mismatch * amend archive * archival logic changed * rename test * archive both proposer and attester seeds * page size 100 * further experiments * further experimentation, archivedProposerIndex seems wrong * test passes * rem log * fix broken test * fix test * gaz * fix imports * ethapis

* Update seed domains (#3872) * Remove Transfers (#3870) * Remove active index roots and compact committee roots (#3869) * Update inclusion reward (#3886) * Alter proposer selection logic (#3884) * Fix early committee bias (#3888) * Remove shards and committees (#3896) * Epoch spec tests v0.9 (#3907) * Block spec test v0.9 (#3905) * rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes * Update deposit contract (#3906) * Proto spec tests v0.9 (#3908) * Starting * Add AggregateAndProof * Unskip block util tests (#3910) * rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes * Unskip block util tests * Slot processing spec test V0.9 (#3912) * Starting * Add AggregateAndProof * Unskip slot processing mainnet test * Unskip minimal spec test for finalization (#3920) * Remove outdated interop tests (#3922) * Rm outdated interop tests * Rm test runner * Gazelle * Update validator to use proposer slot (#3919) * Fix committee assignment (#3931) * Replace shard with committee index (#3930) * Conflict * Clean up (#3933) * Remove shard filter in db (#3936) * Remove lightouse compatibility test (#3939) * Update Committee Cache for v0.9 (#3948) * Updated committee cache * Removed shuffled indices cache * Started testing run time * Lint * Fixed test * Safeguard against nil head state * address edge case * add test * Fixed TestRoundRobinSync by doubling the epochs * Unskip TestProtoCompatability (#3958) * Unskip TestProtoCompatability * Update WORKSPACE * Fix minimal config (#3959) * fix minimal configs * fix hardcoded value in test * Simplify verify att time (#3961) * update readme for deposit contract, regen bindings for vyper 0.1.0b12 (#3963) * update readme for deposit contract, regen bindings * medium * Check nil base state (#3964) * Copy Block When Receiving it From Sync (#3966) * copy block * clone for other service methods too * Change logging of Bitfield (#3956) * change logging of bits * preston's review * Unskip Beacon Server Test (#3962) * run test till the end * fix up proto message types * fmt * resolve broken tests * better error handling * fixing new logic to use archived proposer info * fix up logic * clip using the max effective balance * broken build fix with num arg mismatch * amend archive * archival logic changed * rename test * archive both proposer and attester seeds * page size 100 * further experiments * further experimentation, archivedProposerIndex seems wrong * test passes * rem log * fix broken test * fix test * gaz * fix imports * ethapis * attester server split into subpackage * attester impl split up successfully * validator cleaned up * all packages isolated * include proposer * proper naming * test fix * proper viz * naming * resolved timeout due to config values * init use minimal * added all subfiles * subfile split and gazelle * shards * validator folder * cleanup val * shay feedback

* Update seed domains (#3872) * Remove Transfers (#3870) * Remove active index roots and compact committee roots (#3869) * Update inclusion reward (#3886) * Alter proposer selection logic (#3884) * Fix early committee bias (#3888) * Remove shards and committees (#3896) * Epoch spec tests v0.9 (#3907) * Block spec test v0.9 (#3905) * rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes * Update deposit contract (#3906) * Proto spec tests v0.9 (#3908) * Starting * Add AggregateAndProof * Unskip block util tests (#3910) * rm'ed in protobuf * build proto * build proto * build proto * fix core package * Gazelle * Fixed all the tests * Fixed static test * Comment out spec test for now * One more skip * fix-roundRobinSync (#3862) * Starting but need new seed function * Revert initial sync * Updated Proposer Slashing * Fixed all tests * Lint * Update inclusion reward * Fill randao mixes with eth1 data hash * Test * Fixing test part1 * All tests passing * One last test * Updated config * Build proto * Proper skip message * Conflict and fmt * Removed crosslinks and shards. Built * Format and gazelle * Fixed all the block package tests * Fixed all the helper tests * All epoch package tests pass * All core package tests pass * Fixed operation tests * Started fixing rpc test * RPC tests passed! * Fixed all init sync tests * All tests pass * Fixed blockchain tests * Lint * Lint * Preston's feedback * Starting * Remove container * Fixed block spec tests * All passing except for block_processing test * Failing block processing test * Starting * Add AggregateAndProof * All mainnet test passes * Unskip block util tests * Slot processing spec test V0.9 (#3912) * Starting * Add AggregateAndProof * Unskip slot processing mainnet test * Unskip minimal spec test for finalization (#3920) * Remove outdated interop tests (#3922) * Rm outdated interop tests * Rm test runner * Gazelle * Update validator to use proposer slot (#3919) * Fix committee assignment (#3931) * Replace shard with committee index (#3930) * Conflict * Clean up (#3933) * Remove shard filter in db (#3936) * Remove lightouse compatibility test (#3939) * Update Committee Cache for v0.9 (#3948) * Updated committee cache * Removed shuffled indices cache * Started testing run time * Lint * Fixed test * Safeguard against nil head state * address edge case * add test * Fixed TestRoundRobinSync by doubling the epochs * Unskip TestProtoCompatability (#3958) * Unskip TestProtoCompatability * Update WORKSPACE * Fix minimal config (#3959) * fix minimal configs * fix hardcoded value in test * Simplify verify att time (#3961) * update readme for deposit contract, regen bindings for vyper 0.1.0b12 (#3963) * update readme for deposit contract, regen bindings * medium * Check nil base state (#3964) * Copy Block When Receiving it From Sync (#3966) * copy block * clone for other service methods too * Change logging of Bitfield (#3956) * change logging of bits * preston's review * Unskip Beacon Server Test (#3962) * run test till the end * fix up proto message types * fmt * resolve broken tests * better error handling * fixing new logic to use archived proposer info * fix up logic * clip using the max effective balance * broken build fix with num arg mismatch * amend archive * archival logic changed * rename test * archive both proposer and attester seeds * page size 100 * further experiments * further experimentation, archivedProposerIndex seems wrong * test passes * rem log * fix broken test * fix test * gaz * fix imports * ethapis * setup request/response types for the committees * list beacon committees impl * beacon committees fetch from archive * full list beacon committees implementation * list beacon committees added more useful fields * actually paginate * attester server split into subpackage * attester impl split up successfully * validator cleaned up * all packages isolated * include proposer * proper naming * test fix * proper viz * naming * resolved timeout due to config values * init use minimal * added all subfiles * subfile split and gazelle * shards * validator folder * cleanup val * shay feedback * initial pagination tests passing * paginated tests pass * fix bug regarding total count * pagination tests pass * adding final test, archive * archive test works * regen protos for archival * resolve broken test * test pass * broken archive test * rem helpers * gaz * fix kv test * useful gRPC error code standards * format * resolved bad test * test resolution * ux improvements and bug fixes * complete * comments * Update beacon-chain/archiver/service.go * Update beacon-chain/rpc/beacon/committees.go * elim bad test * preston feedback

terencechain and others added 30 commits October 28, 2019 09:49

rm'ed in protobuf

ad7c4e8

build proto

6eafe7e

build proto

13e05c3

build proto

5fadaa9

fix core package

90c69e7

Gazelle

10ef95e

Fixed all the tests

51729f7

Fixed static test

c80c822

Comment out spec test for now

0008f7e

One more skip

edbffe6

fix-roundRobinSync (#3862)

094f197

Starting but need new seed function

297da47

Sync with 0.9

4fec429

Revert initial sync

908cc4d

Updated Proposer Slashing

b56c0a6

Fixed all tests

b2a6152

Lint

1c319bc

Update inclusion reward

92779ee

Fill randao mixes with eth1 data hash

5161ff1

Test

51accdf

Fixing test part1

9526dd8

All tests passing

75ad52a

One last test

f7ffde9

Updated config

1ba7733

Build proto

98766cc

Merge branch 'v0.9' of https://github.com/prysmaticlabs/prysm into ro…

2939a1e

…ots-removal

Proper skip message

0519929

Conflict and fmt

8936023

Merge with roots removal

02f0165

Merge branch 'proposer-fix' into rm-shards-crosslinks-1

76c5a82

terencechain added the spec-v0.9 label Oct 30, 2019

terencechain self-assigned this Oct 30, 2019

terencechain added 6 commits October 30, 2019 09:21

Fixed operation tests

871c2f4

Started fixing rpc test

5e4faf5

RPC tests passed!

bb647e4

Fixed all init sync tests

14abf56

All tests pass

e4e70c2

Sync with master

3be111a

terencechain changed the base branch from master to v0.9 October 30, 2019 21:27

Fixed blockchain tests

b3558c9

terencechain added 2 commits October 30, 2019 14:54

Lint

5a754dd

Lint

e7300df

terencechain changed the title ~~WIP: Remove shards and committees~~ Remove shards and committees Oct 30, 2019

terencechain added the Ready For Review label Oct 30, 2019

terencechain marked this pull request as ready for review October 30, 2019 22:24

Conflict

cdb7222

prestonvanloon reviewed Oct 30, 2019

View reviewed changes

Preston's feedback

67ab64b

prestonvanloon approved these changes Oct 31, 2019

View reviewed changes

terencechain merged commit c864627 into v0.9 Oct 31, 2019

delete-merged-branch bot deleted the rm-shards-crosslinks-1 branch October 31, 2019 01:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove shards and committees #3896

Remove shards and committees #3896

terencechain commented Oct 30, 2019 •

edited

Loading

codecov bot commented Oct 30, 2019 •

edited

Loading

prestonvanloon Oct 30, 2019

terencechain Oct 30, 2019

prestonvanloon Oct 30, 2019

terencechain Oct 30, 2019

prestonvanloon Oct 30, 2019

terencechain Oct 31, 2019

prestonvanloon Oct 30, 2019

terencechain Oct 31, 2019

prestonvanloon Oct 30, 2019

prestonvanloon Oct 30, 2019

	t.Errorf("SlotsPerEpoch should be 64 for these tests to pass")
	t.Errorf("SlotsPerEpoch should be 32 for these tests to pass")

Remove shards and committees #3896

Remove shards and committees #3896

Conversation

terencechain commented Oct 30, 2019 • edited Loading

codecov bot commented Oct 30, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terencechain commented Oct 30, 2019 •

edited

Loading

codecov bot commented Oct 30, 2019 •

edited

Loading