Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Offchain Phragmén BREAKING. #4517

Merged
merged 144 commits into from
Mar 26, 2020
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
144 commits
Select commit Hold shift + click to select a range
46ecd58
Initial skeleton for offchain phragmen
kianenigma Jan 2, 2020
a830ec9
Basic compact encoding decoding for results
kianenigma Jan 6, 2020
8448ee9
add compact files
kianenigma Jan 6, 2020
5e26a68
Bring back Self::ensure_storage_upgraded();
kianenigma Jan 6, 2020
65b2df6
Make staking use compact stuff.
kianenigma Jan 7, 2020
85b726c
First seemingly working version of reduce, full of todos
kianenigma Jan 13, 2020
bf053c2
Master.into()
kianenigma Jan 13, 2020
ca68e24
Everything phragmen related works again.
kianenigma Jan 13, 2020
62e61db
Signing made easier, still issues.
kianenigma Jan 14, 2020
c4a77ac
Signing from offchain compile fine 😎
kianenigma Jan 15, 2020
31b699a
make compact work with staked asssignment
kianenigma Jan 15, 2020
c5d8c51
Evaluation basics are in place.
kianenigma Jan 15, 2020
6d4494e
Master.into()
kianenigma Jan 15, 2020
35a7872
Move reduce into crate. Document stuff
kianenigma Jan 15, 2020
0540646
move reduce into no_std
kianenigma Jan 16, 2020
da66da2
Add files
kianenigma Jan 16, 2020
d95010d
Remove other std deps. Runtime compiles
kianenigma Jan 16, 2020
3f23367
Seemingly it is al stable; cycle implemented but not integrated.
kianenigma Jan 20, 2020
3cb47fb
Add fuzzing code.
kianenigma Jan 21, 2020
a21e90b
Cleanup reduce a bit more.
kianenigma Jan 21, 2020
df7676b
a metric ton of tests for staking; wip 🔨
kianenigma Jan 22, 2020
6f4a2fe
Master.into()
kianenigma Jan 22, 2020
889b18c
Implement a lot more of the tests.
kianenigma Jan 23, 2020
50f5399
wip getting the unsigned stuff to work
kianenigma Jan 23, 2020
2bbdec4
A bit gleanup for unsigned debug
kianenigma Jan 24, 2020
c9cfcb8
Clean and finalize compact code.
kianenigma Jan 24, 2020
3633a77
Document reduce.
kianenigma Jan 27, 2020
686d263
Still problems with signing
kianenigma Jan 27, 2020
295cdd7
We officaly duct taped the transaction submission stuff. 🤓
kianenigma Jan 27, 2020
57b2b4e
Deadlock with keys again
kianenigma Jan 28, 2020
6a33fc1
Runtime builds
kianenigma Jan 28, 2020
4e951af
Unsigned test works 🙌
kianenigma Jan 28, 2020
5ab3cf0
Some cleanups
kianenigma Jan 28, 2020
c88df34
Master.into()
kianenigma Jan 28, 2020
31fd60e
Make all the tests compile and stuff
kianenigma Jan 29, 2020
2623d98
Minor cleanup
kianenigma Feb 3, 2020
c911a87
Master.into()
kianenigma Feb 3, 2020
2ed0a1d
fix more merge stuff
kianenigma Feb 3, 2020
6bef2d9
Most tests work again.
kianenigma Feb 5, 2020
201c11c
a very nasty bug in reduce
kianenigma Feb 6, 2020
eef09e5
Fix all integrations
kianenigma Feb 6, 2020
51fc865
Fix more todos
kianenigma Feb 6, 2020
aa6186c
Revamp everything and everything
kianenigma Feb 11, 2020
596bc77
Master.into()
kianenigma Feb 11, 2020
084a355
Remove bogus test
kianenigma Feb 11, 2020
817f8aa
Some review grumbles.
kianenigma Feb 11, 2020
d1e290f
Make it work with custom key type
kianenigma Feb 12, 2020
02c4e3b
Some fixes
kianenigma Feb 12, 2020
aea7ad2
Fix doc test
kianenigma Feb 12, 2020
e2210cc
loop for submission
kianenigma Feb 13, 2020
ca20033
Fix cli, keyring etc.
kianenigma Feb 13, 2020
b2c4416
Merged with the generic phragmen -- maybe not perfectly but works
kianenigma Feb 13, 2020
10e97e3
some cleanup
kianenigma Feb 14, 2020
19cf777
Master.into()
kianenigma Feb 14, 2020
54a5a35
Fix staking tests again
kianenigma Feb 14, 2020
d746fd4
fix per-things; bring patches from benchmarking
kianenigma Feb 18, 2020
2f75ea7
better score prediction
kianenigma Feb 18, 2020
2fa064b
Add fuzzer, more patches.
kianenigma Feb 19, 2020
0afbde3
Some fixes
kianenigma Feb 19, 2020
16d8c64
Fix tests
kianenigma Feb 19, 2020
b8efef4
More docs
kianenigma Feb 20, 2020
e72d642
Remove unused generics
kianenigma Feb 20, 2020
9f227f9
Remove max-nominator footgun
kianenigma Feb 20, 2020
9116285
Better fuzzer
kianenigma Feb 21, 2020
36bbf4b
Master.into()
kianenigma Feb 21, 2020
44c0a6b
Disable it ❌
kianenigma Feb 21, 2020
c3769fa
Master.into()
kianenigma Feb 21, 2020
f21df59
Bump.
kianenigma Feb 21, 2020
f9f92c4
Another round of self-review
kianenigma Feb 26, 2020
4df9ed4
Master.into()
kianenigma Feb 26, 2020
037057c
Refactor a lot
kianenigma Feb 27, 2020
2b89c0e
More major fixes in perThing
kianenigma Feb 28, 2020
4e44501
Add new fuzz file
kianenigma Feb 28, 2020
5c3fc6c
Master.into()
kianenigma Feb 28, 2020
7287c2b
Update lock
kianenigma Feb 28, 2020
ac6ce78
fix fuzzing code.
kianenigma Feb 28, 2020
7be2268
Fix nominator retain test
kianenigma Mar 2, 2020
1bd7850
Add slashing check
kianenigma Mar 2, 2020
5f9844d
Update frame/staking/src/tests.rs
kianenigma Mar 2, 2020
555007f
Some formatting nits
kianenigma Mar 2, 2020
c16ad73
Merge branch 'kiz-offchain-phragmen-4' of github.com:paritytech/subst…
kianenigma Mar 2, 2020
c8cc72f
Review comments.
kianenigma Mar 3, 2020
ceef0e6
Merge branch 'master' of github.com:paritytech/substrate into kiz-off…
kianenigma Mar 3, 2020
62d4f5e
Fix cargo file
kianenigma Mar 3, 2020
fd9816a
Master.into()
kianenigma Mar 4, 2020
85f1aa4
Almost all tests work again
kianenigma Mar 4, 2020
9b8af93
Update frame/staking/src/tests.rs
kianenigma Mar 4, 2020
58fcf27
Fix review comments
kianenigma Mar 4, 2020
7f47696
Merge branch 'kiz-offchain-phragmen-4' of github.com:paritytech/subst…
kianenigma Mar 4, 2020
ef7eab2
More review stuff
kianenigma Mar 4, 2020
5f70434
Some nits
kianenigma Mar 4, 2020
4d82b4d
Merge branch 'master' of github.com:paritytech/substrate into kiz-off…
kianenigma Mar 4, 2020
58267dc
Fix new staking / session / babe relation
kianenigma Mar 5, 2020
bee2f5d
Update primitives/phragmen/src/lib.rs
kianenigma Mar 5, 2020
3def57b
Update primitives/phragmen/src/lib.rs
kianenigma Mar 5, 2020
9590986
Update primitives/phragmen/compact/src/lib.rs
kianenigma Mar 5, 2020
6566d81
Some doc updates to slashing
kianenigma Mar 5, 2020
b01000e
Merge branch 'kiz-offchain-phragmen-4' of github.com:paritytech/subst…
kianenigma Mar 5, 2020
1667f6e
Fix derive
kianenigma Mar 5, 2020
155e9a3
Master.into()
kianenigma Mar 5, 2020
e25c103
Remove imports
kianenigma Mar 5, 2020
81ddd48
Remove unimplemented tests
kianenigma Mar 6, 2020
87bb4d1
Master.into()
kianenigma Mar 9, 2020
eda82ae
nits
kianenigma Mar 9, 2020
56d1f5e
Remove dbg
kianenigma Mar 9, 2020
0c9c20f
Better fuzzing params
kianenigma Mar 9, 2020
f308817
Master.into()
kianenigma Mar 9, 2020
f014e2c
Remove unused pref map
kianenigma Mar 9, 2020
26d9329
Deferred Slashing/Offence for offchain Phragmen (#5151)
kianenigma Mar 10, 2020
ecd042e
Fix build
kianenigma Mar 10, 2020
63eb8c4
review comments
kianenigma Mar 12, 2020
3cd2c55
fix more
kianenigma Mar 12, 2020
1e4291c
Merge branch 'master' of github.com:paritytech/substrate into kiz-off…
kianenigma Mar 12, 2020
918e58e
fix build
kianenigma Mar 12, 2020
537d6ee
Some cleanups and self-reviews
kianenigma Mar 13, 2020
98f9c1b
More minor self reviews
kianenigma Mar 13, 2020
b553cf5
Master.into()
kianenigma Mar 13, 2020
199d8fb
Final nits
kianenigma Mar 13, 2020
a424a94
Merge branch 'master' of github.com:paritytech/substrate into kiz-off…
kianenigma Mar 13, 2020
42dbf06
A holy darn big upstream merge.
kianenigma Mar 19, 2020
202db20
Some merge fixes.
kianenigma Mar 19, 2020
8372453
opt comment
kianenigma Mar 20, 2020
db50363
Master.into()
kianenigma Mar 20, 2020
2692605
Fix build
kianenigma Mar 20, 2020
2cf090c
Fix build again.
kianenigma Mar 20, 2020
ffda508
Merge branch 'master' into kiz-offchain-phragmen-4
gavofyork Mar 21, 2020
4c56e36
Master.into()
kianenigma Mar 21, 2020
18558af
Update frame/staking/fuzz/fuzz_targets/submit_solution.rs
kianenigma Mar 22, 2020
6f9a2c5
Update frame/staking/src/slashing.rs
kianenigma Mar 23, 2020
cd3df80
Update frame/staking/src/offchain_election.rs
kianenigma Mar 23, 2020
1153084
Fix review comments
kianenigma Mar 23, 2020
4c274ab
Merge branch 'kiz-offchain-phragmen-4' of github.com:paritytech/subst…
kianenigma Mar 23, 2020
c75f441
Master.into()
kianenigma Mar 23, 2020
34e3d7b
fix test
kianenigma Mar 23, 2020
ee3c4cb
=== 🔑 Revamp without staking key.
kianenigma Mar 24, 2020
3c5dd95
Master.into()
kianenigma Mar 24, 2020
5a1aa2f
Master.into()
kianenigma Mar 25, 2020
03bb2a9
final round of changes.
kianenigma Mar 26, 2020
8dd2564
Master.into()
kianenigma Mar 26, 2020
4514432
Fix cargo-deny
kianenigma Mar 26, 2020
ce2ff51
Merge branch 'master' into kiz-offchain-phragmen-4
gavofyork Mar 26, 2020
9c0f42c
Master.into()
kianenigma Mar 26, 2020
94efcff
Master.into()
kianenigma Mar 26, 2020
8182947
Update frame/staking/src/lib.rs
kianenigma Mar 26, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions bin/node/runtime/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -439,6 +439,7 @@ impl pallet_sudo::Trait for Runtime {
type SubmitTransaction = TransactionSubmitter<ImOnlineId, Runtime, UncheckedExtrinsic>;

parameter_types! {
// assume 1 slot == 1 block
pub const SessionDuration: BlockNumber = EPOCH_DURATION_IN_SLOTS as _;
}

Expand Down
4 changes: 0 additions & 4 deletions frame/babe/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ authors = ["Parity Technologies <admin@parity.io>"]
edition = "2018"

[dependencies]
hex-literal = "0.2.1"
codec = { package = "parity-scale-codec", version = "1.0.0", default-features = false, features = ["derive"] }
serde = { version = "1.0.101", optional = true }
sp-inherents = { version = "2.0.0", default-features = false, path = "../../primitives/inherents" }
Expand All @@ -21,11 +20,8 @@ sp-consensus-babe = { version = "0.8", default-features = false, path = "../../p
sp-io ={ path = "../../primitives/io", default-features = false }

[dev-dependencies]
lazy_static = "1.4.0"
parking_lot = "0.9.0"
sp-version = { version = "2.0.0", default-features = false, path = "../../primitives/version" }
sp-core = { version = "2.0.0", path = "../../primitives/core" }
substrate-test-runtime = { version = "2.0.0", path = "../../test-utils/runtime" }

[features]
default = ["std"]
Expand Down
34 changes: 28 additions & 6 deletions frame/babe/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ use sp_std::{result, prelude::*};
use frame_support::{decl_storage, decl_module, traits::FindAuthor, traits::Get};
use sp_timestamp::OnTimestampSet;
use sp_runtime::{generic::DigestItem, ConsensusEngineId, Perbill};
use sp_runtime::traits::{IsMember, SaturatedConversion, Saturating, RandomnessBeacon};
use sp_runtime::traits::{IsMember, SaturatedConversion, Saturating, RandomnessBeacon, One, Bounded};
use sp_staking::{
SessionIndex,
offence::{Offence, Kind},
Expand Down Expand Up @@ -305,12 +305,31 @@ impl<T: Trait> Module<T> {
// epoch 0 as having started at the slot of block 1. We want to use
// the same randomness and validator set as signalled in the genesis,
// so we don't rotate the epoch.
now != sp_runtime::traits::One::one() && {
now != One::one() && {
let diff = CurrentSlot::get().saturating_sub(Self::current_epoch_start());
diff >= T::EpochDuration::get()
}
}

/// Return the _best guess_ block number, at which the next epoch change is predicted to happen.
///
/// In other word, this is only accurate if no slots are missed. Given missed slots, the slot
/// number will grow while the block number will not. Hence, the result can be interpreted as an
/// upper bound.
///
/// -------------- IMPORTANT NOTE --------------
/// This implementation is linked to how [`should_epoch_change`] is working. This might need to
kianenigma marked this conversation as resolved.
Show resolved Hide resolved
/// be updated accordingly, if the underlying mechanics of slot and epochs change.
kianenigma marked this conversation as resolved.
Show resolved Hide resolved
pub fn next_epoch_change(now: T::BlockNumber) -> T::BlockNumber {
kianenigma marked this conversation as resolved.
Show resolved Hide resolved
let next_slot = Self::current_epoch_start().saturating_add(T::EpochDuration::get());
let slots_remaining = next_slot
.checked_sub(CurrentSlot::get())
.unwrap_or(Bounded::max_value());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.unwrap_or(Bounded::max_value());
.unwrap_or(0);

Wouldn't this make more sense?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes more sense given the caller. If we return zero, if the caller is comparing this value to be le of some threshold, then it will always think that the condition is met. While by returning max_value we say to the caller infinite blocks are remaining, so that no condition is triggered.

But this is tailored for the staking use case. I will change to to returning just some option.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had this exact same question. Option does seem better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes I marked this as resolved by mistake. Will refactor, option is better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not done.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought a bit more about it.

if checked_sub fails here, it means a very severe bug in babe is happening. I will add a warning log to it, but returning None seems like a bit ignorant as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nonetheless, finally done.

// This is a best effort guess. Drifts in the slot/block ratio will cause errors here.
let blocks_remaining: T::BlockNumber = slots_remaining.saturated_into();
now.saturating_add(blocks_remaining)
}

/// DANGEROUS: Enact an epoch change. Should be done on every block where `should_epoch_change` has returned `true`,
/// and the caller is the only caller of this function.
///
Expand All @@ -322,10 +341,7 @@ impl<T: Trait> Module<T> {
) {
// PRECONDITION: caller has done initialization and is guaranteed
// by the session module to be called before this.
#[cfg(debug_assertions)]
{
assert!(Self::initialized().is_some())
}
debug_assert!(Self::initialized().is_some());

// Update epoch index
let epoch_index = EpochIndex::get()
Expand Down Expand Up @@ -471,6 +487,12 @@ impl<T: Trait> OnTimestampSet<T::Moment> for Module<T> {
fn on_timestamp_set(_moment: T::Moment) { }
}

impl<T: Trait> frame_support::traits::PredictNextSessionChange<T::BlockNumber> for Module<T> {
fn predict_next_session_change(now: T::BlockNumber) -> T::BlockNumber {
kianenigma marked this conversation as resolved.
Show resolved Hide resolved
Self::next_epoch_change(now)
}
}

impl<T: Trait> sp_runtime::BoundToRuntimeAppPublic for Module<T> {
type Public = AuthorityId;
}
Expand Down
51 changes: 45 additions & 6 deletions frame/babe/src/mock.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,16 @@
//! Test utilities
#![allow(dead_code, unused_imports)]

use super::{Trait, Module, GenesisConfig};
use codec::Encode;
use super::{Trait, Module, GenesisConfig, CurrentSlot};
use sp_consensus_babe::AuthorityId;
use sp_runtime::{
traits::IdentityLookup, Perbill, testing::{Header, UintAuthorityId}, impl_opaque_keys,
Perbill, impl_opaque_keys,
testing::{Header, UintAuthorityId, Digest, DigestItem},
traits::{IdentityLookup, OnInitialize},
};
use sp_version::RuntimeVersion;
use frame_support::{impl_outer_origin, parameter_types, weights::Weight};
use frame_support::{impl_outer_origin, parameter_types, StorageValue, weights::Weight};
use sp_io;
use sp_core::{H256, Blake2Hasher};

Expand All @@ -45,7 +48,6 @@ parameter_types! {
pub const MinimumPeriod: u64 = 1;
pub const EpochDuration: u64 = 3;
pub const ExpectedBlockTime: u64 = 1;
pub const Version: RuntimeVersion = substrate_test_runtime::VERSION;
pub const DisabledValidatorsThreshold: Perbill = Perbill::from_percent(16);
}

Expand All @@ -55,7 +57,7 @@ impl frame_system::Trait for Test {
type BlockNumber = u64;
type Call = ();
type Hash = H256;
type Version = Version;
type Version = ();
kianenigma marked this conversation as resolved.
Show resolved Hide resolved
type Hashing = sp_runtime::traits::BlakeTwo256;
type AccountId = DummyValidatorId;
type Lookup = IdentityLookup<Self::AccountId>;
Expand All @@ -78,7 +80,7 @@ impl pallet_session::Trait for Test {
type Event = ();
type ValidatorId = <Self as frame_system::Trait>::AccountId;
type ShouldEndSession = Babe;
type SessionHandler = (Babe,Babe,);
type SessionHandler = (Babe,);
kianenigma marked this conversation as resolved.
Show resolved Hide resolved
type OnSessionEnding = ();
type ValidatorIdOf = ();
type SelectInitialValidators = ();
Expand Down Expand Up @@ -106,5 +108,42 @@ pub fn new_test_ext(authorities: Vec<DummyValidatorId>) -> sp_io::TestExternalit
t.into()
}

pub fn go_to_block(n: u64, s: u64) {
let pre_digest = make_pre_digest(0, s, [1; 32], [0xff; 64]);
System::initialize(&n, &Default::default(), &Default::default(), &pre_digest);
System::set_block_number(n);
if s > 1 {
CurrentSlot::put(s);
}
// includes a call into `Babe::do_initialize`.
Session::on_initialize(n);
}

/// Slots will grow accordingly to blocks
pub fn progress_to_block(n: u64) {
let mut slot = Babe::current_slot() + 1;
for i in System::block_number()+1..=n {
go_to_block(i, slot);
slot += 1;
}
}

pub fn make_pre_digest(
authority_index: sp_consensus_babe::AuthorityIndex,
slot_number: sp_consensus_babe::SlotNumber,
vrf_output: [u8; sp_consensus_babe::VRF_OUTPUT_LENGTH],
vrf_proof: [u8; sp_consensus_babe::VRF_PROOF_LENGTH],
) -> Digest {
let digest_data = sp_consensus_babe::RawBabePreDigest::Primary {
authority_index,
slot_number,
vrf_output,
vrf_proof,
};
let log = DigestItem::PreRuntime(sp_consensus_babe::BABE_ENGINE_ID, digest_data.encode());
Digest { logs: vec![log] }
}

pub type System = frame_system::Module<Test>;
pub type Babe = Module<Test>;
pub type Session = pallet_session::Module<Test>;
41 changes: 23 additions & 18 deletions frame/babe/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@
//! Consensus extension module tests for BABE consensus.

use super::*;
use mock::{new_test_ext, Babe, Test};
use sp_runtime::{traits::OnFinalize, testing::{Digest, DigestItem}};
use mock::*;
use sp_runtime::{traits::OnFinalize};
use pallet_session::ShouldEndSession;

const EMPTY_RANDOMNESS: [u8; 32] = [
Expand All @@ -28,22 +28,6 @@ const EMPTY_RANDOMNESS: [u8; 32] = [
217, 153, 138, 37, 48, 192, 248, 0,
];

fn make_pre_digest(
authority_index: sp_consensus_babe::AuthorityIndex,
slot_number: sp_consensus_babe::SlotNumber,
vrf_output: [u8; sp_consensus_babe::VRF_OUTPUT_LENGTH],
vrf_proof: [u8; sp_consensus_babe::VRF_PROOF_LENGTH],
) -> Digest {
let digest_data = sp_consensus_babe::RawBabePreDigest::Primary {
authority_index,
slot_number,
vrf_output,
vrf_proof,
};
let log = DigestItem::PreRuntime(sp_consensus_babe::BABE_ENGINE_ID, digest_data.encode());
Digest { logs: vec![log] }
}

#[test]
fn empty_randomness_is_correct() {
let s = compute_randomness([0; RANDOMNESS_LENGTH], 0, std::iter::empty(), None);
Expand Down Expand Up @@ -124,3 +108,24 @@ fn authority_index() {
"Trivially invalid authorities are ignored")
})
}

#[test]
fn can_predict_next_epoch_change() {
new_test_ext(vec![]).execute_with(|| {
assert_eq!(<Test as Trait>::EpochDuration::get(), 3);
// this sets the genesis slot to 6;
go_to_block(1, 6);
assert_eq!(Babe::genesis_slot(), 6);
assert_eq!(Babe::current_slot(), 6);
assert_eq!(Babe::epoch_index(), 0);

progress_to_block(5);

assert_eq!(Babe::epoch_index(), 5 / 3);
assert_eq!(Babe::current_slot(), 10);

// next epoch change will be at
assert_eq!(Babe::current_epoch_start(), 9); // next change will be 12, 2 slots from now
assert_eq!(Babe::next_epoch_change(System::block_number()), 5 + 2);
})
}
Loading