Re-built tendermint-proto with serialization annotations (#639)

* Re-built tendermint-proto with serialization annontations

* Bumped tendermint-proto version number

* AbciInfo reimplemented as domain type with JSON serialization

* JSON serialization work

* Block serialization work

* Block JSON encoding fixed

* Add test for JSON timestamp de/serialization precision

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Fix Go/chrono timestamp serialization compatibility

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Tendermint basic integration tests succeed, clippy fixes

* part_set_header fix and others

* More cleanup of serialization

* txs fix and BlockMeta serialization

* Test fixes, serialization fixes

* tendermint integration tests all succeed

* More test fixes

* More test fixes

* More test fixes

* model-based tests updated

* ConsensusParams domain type

* genesis temp fix

* Removed single_step tests as mentioned by Andrey

* Merge conflict fix refix, voting_power tests removed as per Andrey

* Added Hash custom serialization

* Fixes some bisection test files

* Restored voting_power tests

* More test fixes

* fmt fix

* Disabled model-based single step tests until they can be recreated correctly

* Enable alias for part_set_header

Right now we need to allow for compatibility with the latest version of
Tendermint, as well as v0.34.0-rc5. Between these versions, the `parts`
field was renamed to `part_set_header` in the JSON serialization. We
need to support both field names deserializing into the same data
structures.

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Add Anca's changes and fix test failures

I've added some changes that Anca made and have tweaked a few tests to
get them to pass (the serialization was breaking, so I fixed the
fixtures).

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* cargo fmt

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Provide default validator set if none supplied in genesis

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* add Eq to TrustThresholdFraction

* Fix git clone/update mechanism for proto compiler

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Implement domain types for merke Proof

* Add checks for abci_query proof deserialization

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Make the consensus parameter's version Option

* Add alias "proofOps" for "proof" field in abci_query response

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Block TryFrom cleaning

* Make structs with constructors non-exhaustive

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Update CHANGELOG

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Use simple constructor for validator set construction

Signed-off-by: Thane Thomson <connect@thanethomson.com>

* Restore getters for Block and SignedHeader

Signed-off-by: Thane Thomson <connect@thanethomson.com>

Co-authored-by: Thane Thomson <connect@thanethomson.com>
Co-authored-by: Anca Zamfir <zamfiranca@gmail.com>

CHANGELOG.md

@@ -1,5 +1,14 @@
 ## Unreleased
 
+### BREAKING CHANGES:
+
+- `[tendermint]` - Direct serialization capabilities have been removed from the
+  domain types. They are temporarily available in the `protos` crate. **NB:
+  this is unstable and is planned to change again in v0.17.0-rc3**. ([#639])
+- `[tendermint]` - Work has started on making it compulsory to construct domain
+  types by way of their constructors to ensure validity. This work is scheduled
+  for completion in v0.17.0-rc3. ([#639])
+
 ### BUG FIXES:
 
 - `[light-client]` Fix bug where a commit with only absent signatures would be
@@ -9,6 +18,7 @@
 
 [#650]: https://github.com/informalsystems/tendermint-rs/issues/650
 [#652]: https://github.com/informalsystems/tendermint-rs/pulls/652
+[#639]: https://github.com/informalsystems/tendermint-rs/pull/639
 
 ## v0.17.0-rc1
 

light-client/src/builder/light_client.rs

@@ -134,7 +134,7 @@ impl LightClientBuilder<NoTrustedState> {
         let trusted_state = self
             .light_store
             .latest_trusted_or_verified()
-            .ok_or_else(|| error::Kind::NoTrustedStateInStore)?;
+            .ok_or(error::Kind::NoTrustedStateInStore)?;
 
         self.trust_light_block(trusted_state)
     }

light-client/src/components/io.rs

@@ -155,7 +155,7 @@ mod prod {
             let res = block_on(self.timeout, async move { client.validators(height).await })?;
 
             match res {
-                Ok(response) => Ok(TMValidatorSet::new(response.validators)),
+                Ok(response) => Ok(TMValidatorSet::new_simple(response.validators)),
                 Err(err) => Err(IoError::RpcError(err)),
             }
         }

light-client/src/light_client.rs

@@ -173,7 +173,7 @@ impl LightClient {
             let trusted_state = state
                 .light_store
                 .latest_trusted_or_verified()
-                .ok_or_else(|| ErrorKind::NoInitialTrustedState)?;
+                .ok_or(ErrorKind::NoInitialTrustedState)?;
 
             if target_height < trusted_state.height() {
                 bail!(ErrorKind::TargetLowerThanTrustedState {

light-client/src/operations/voting_power.rs

@@ -10,7 +10,7 @@ use serde::{Deserialize, Serialize};
 use std::collections::HashSet;
 use std::fmt;
 
-use std::convert::{TryFrom, TryInto};
+use std::convert::TryFrom;
 use tendermint::block::CommitSig;
 use tendermint::trust_threshold::TrustThreshold as _;
 use tendermint::vote::{SignedVote, ValidatorIndex, Vote};
@@ -211,7 +211,7 @@ fn non_absent_vote(
     Some(Vote {
         vote_type: tendermint::vote::Type::Precommit,
         height: commit.height,
-        round: commit.round.try_into().unwrap(),
+        round: commit.round,
         block_id,
         timestamp: Some(timestamp),
         validator_address,

light-client/src/predicates/errors.rs

@@ -16,8 +16,10 @@ pub enum VerificationError {
     #[error("header from the future: header_time={header_time} now={now}")]
     HeaderFromTheFuture {
         /// Time in the header
+        #[serde(with = "tendermint::serializers::time")]
         header_time: Time,
         /// Current time
+        #[serde(with = "tendermint::serializers::time")]
         now: Time,
     },
 
@@ -52,26 +54,32 @@ pub enum VerificationError {
     #[error("invalid commit value: header_hash={header_hash} commit_hash={commit_hash}")]
     InvalidCommitValue {
         /// Header hash
+        #[serde(with = "tendermint::serializers::hash")]
         header_hash: Hash,
         /// Commit hash
+        #[serde(with = "tendermint::serializers::hash")]
         commit_hash: Hash,
     },
 
     /// Hash mismatch for the next validator set
     #[error("invalid next validator set: header_next_validators_hash={header_next_validators_hash} next_validators_hash={next_validators_hash}")]
     InvalidNextValidatorSet {
         /// Next validator set hash
+        #[serde(with = "tendermint::serializers::hash")]
         header_next_validators_hash: Hash,
         /// Validator set hash
+        #[serde(with = "tendermint::serializers::hash")]
         next_validators_hash: Hash,
     },
 
     /// Hash mismatch for the validator set
     #[error("invalid validator set: header_validators_hash={header_validators_hash} validators_hash={validators_hash}")]
     InvalidValidatorSet {
         /// Hash of validator set stored in header
+        #[serde(with = "tendermint::serializers::hash")]
         header_validators_hash: Hash,
         /// Actual hash of validator set in header
+        #[serde(with = "tendermint::serializers::hash")]
         validators_hash: Hash,
     },
 

light-client/src/supervisor.rs

@@ -216,7 +216,7 @@ impl Supervisor {
             Ok(verified_block) => {
                 let trusted_block = primary
                     .latest_trusted()
-                    .ok_or_else(|| ErrorKind::NoTrustedState(Status::Trusted))?;
+                    .ok_or(ErrorKind::NoTrustedState(Status::Trusted))?;
 
                 // Perform fork detection with the highest verified block and the trusted block.
                 let outcome = self.detect_forks(&verified_block, &trusted_block)?;

light-client/src/tests.rs

@@ -38,6 +38,7 @@ pub struct Initial {
     pub signed_header: SignedHeader,
     pub next_validator_set: ValidatorSet,
     pub trusting_period: DurationStr,
+    #[serde(with = "tendermint::serializers::time")]
     pub now: Time,
 }
 
@@ -48,6 +49,7 @@ pub struct TestBisection<LB> {
     pub primary: Provider<LB>,
     pub witnesses: Vec<WitnessProvider<LB>>,
     pub height_to_verify: HeightStr,
+    #[serde(with = "tendermint::serializers::time")]
     pub now: Time,
     pub expected_output: Option<String>,
     pub expected_num_of_bisections: usize,

light-client/src/types.rs

@@ -127,9 +127,11 @@ pub struct LatestStatus {
     /// The latest height we are trusting.
     pub height: Option<u64>,
     /// The latest block hash we are trusting.
+    #[serde(with = "tendermint::serializers::option_hash")]
     pub block_hash: Option<Hash>,
     /// The latest validator set we are trusting.
     /// Note that this potentially did not yet sign a header yet.
+    #[serde(with = "tendermint::serializers::option_hash")]
     pub valset_hash: Option<Hash>,
     /// The list of fullnodes we are connected to, primary and witnesses.
     pub connected_nodes: Vec<PeerId>,

light-client/tests/model_based.rs

@@ -3,7 +3,6 @@ use serde::{Deserialize, Serialize};
 use std::convert::TryFrom;
 use std::str::FromStr;
 use std::time::Duration;
-use tendermint::block::CommitSigs;
 use tendermint_light_client::components::verifier::Verdict;
 use tendermint_light_client::types::ValidatorSet;
 use tendermint_light_client::{
@@ -63,6 +62,7 @@ pub struct SingleStepTestCase {
 pub struct BlockVerdict {
     block: AnonLightBlock,
     testgen_block: TestgenLightBlock,
+    #[serde(with = "tendermint::serializers::time")]
     now: Time,
     verdict: LiteVerdict,
 }
@@ -230,7 +230,7 @@ impl SingleStepTestFuzzer for HeaderValHashFuzzer {
             Validator::new("2"),
             Validator::new("3"),
         ];
-        let valset = ValidatorSet::new(generate_validators(&vals).unwrap());
+        let valset = ValidatorSet::new_simple(generate_validators(&vals).unwrap());
 
         input.block.validators = valset;
         (String::from("header validators_hash"), true)
@@ -245,7 +245,7 @@ impl SingleStepTestFuzzer for HeaderNextValHashFuzzer {
             Validator::new("2"),
             Validator::new("3"),
         ];
-        let valset = ValidatorSet::new(generate_validators(&vals).unwrap());
+        let valset = ValidatorSet::new_simple(generate_validators(&vals).unwrap());
 
         input.block.next_validators = valset;
         (String::from("header next_validators_hash"), true)
@@ -319,12 +319,12 @@ struct CommitRoundFuzzer {}
 impl SingleStepTestFuzzer for CommitRoundFuzzer {
     fn fuzz_input(input: &mut BlockVerdict) -> (String, bool) {
         let mut rng = rand::thread_rng();
-        let r: u32 = input.block.signed_header.commit.round;
+        let r: u32 = input.block.signed_header.commit.round.value();
         let mut round: u32 = rng.gen();
         while round == r {
             round = rng.gen();
         }
-        input.block.signed_header.commit.round = round;
+        input.block.signed_header.commit.round = (round as u16).into();
         (format!("commit round from {} into {}", r, round), true)
     }
 }
@@ -425,8 +425,7 @@ impl SingleStepTestFuzzer for SignaturesFuzzer {
 
             input.block.signed_header.commit = commit.generate().unwrap();
         } else {
-            let commitsigs = CommitSigs::new(vec![]);
-            input.block.signed_header.commit.signatures = commitsigs;
+            input.block.signed_header.commit.signatures = vec![];
         }
 
         (String::from("signatures"), true)
@@ -630,6 +629,7 @@ fn model_based_test_batch(batch: ApalacheTestBatch) -> Vec<(String, String)> {
 const TEST_DIR: &str = "./tests/support/model_based";
 
 #[test]
+#[ignore]
 fn run_model_based_single_step_tests() {
     let mut tester = Tester::new("test_run", TEST_DIR);
     tester.add_test_with_env("static model-based single-step test", fuzz_single_step_test);