Add new line after writing block to hex file. #10984

seunlanlege · 2019-08-21T16:08:38Z

Previously when you exported your DB in hex format, it appends all block RLP on the same line. So when you tried to import, you'd only be able to import 1 block.

This pr adds a new line after every block RLP, and also introduces ImportBlocks and ExportBlocks trait in order to allow for the core functionality of importing/exporting to be tested.

closes #10942

dvdplm

A test for this?

Demi-Marie

Partial reads need to be handled.

ethcore/blockchain/src/blockchain.rs

ethcore/src/client/client.rs

ethcore/client-traits/src/lib.rs

ethcore/src/client/client.rs

parity/blockchain.rs

ordian

LGTM

ethcore/blockchain/src/blockchain.rs

ethcore/client-traits/src/lib.rs

ethcore/src/client/client.rs

ethcore/src/tests/client.rs

dvdplm

Lovely. Minor nits.

ethcore/client-traits/src/lib.rs

ethcore/src/client/client.rs

not in scope

dvdplm · 2019-09-06T11:09:47Z

@seunlanlege can you resolve merge conflicts so we can get this one in?

…Blocks

Co-Authored-By: David <dvdplm@gmail.com>

dvdplm · 2019-09-07T21:33:19Z

@seunlanlege looks like the merge was not successful. PTAL.

@demimarie-parity is your review still current here?

…he-right-place * master: cleanup json crate (#11027) [spec] add istanbul test spec (#11033) [json-spec] make blake2 pricing spec more readable (#11034) Add blake2_f precompile (#11017) Add new line after writing block to hex file. (#10984) fix: remove unused error-chain (#11028) fix: remove needless use of itertools (#11029) Convert `std::test` benchmarks to use Criterion (#10999) Fix block detail updating (#11015) [trace] introduce trace failed to Ext (#11019) cli: update usage and version headers (#10924) [private-tx] remove unused rand (#11024)

* master: (70 commits) ethcore: remove `test-helper feat` from build (#11047) Include test-helpers from ethjson (#11045) [ethcore]: cleanup dependencies (#11043) add more tx tests (#11038) Fix parallel transactions race-condition (#10995) [ethcore]: make it compile without `test-helpers` feature (#11036) Benchmarks for block verification (#11035) Move snapshot related traits to their proper place (#11012) cleanup json crate (#11027) [spec] add istanbul test spec (#11033) [json-spec] make blake2 pricing spec more readable (#11034) Add blake2_f precompile (#11017) Add new line after writing block to hex file. (#10984) fix: remove unused error-chain (#11028) fix: remove needless use of itertools (#11029) Convert `std::test` benchmarks to use Criterion (#10999) Fix block detail updating (#11015) [trace] introduce trace failed to Ext (#11019) cli: update usage and version headers (#10924) [private-tx] remove unused rand (#11024) ...

dvdplm · 2019-10-27T20:50:29Z

This would have been good to backport sooner. :/

dvdplm · 2019-10-27T21:00:14Z

@seunlanlege do you think you could take care of that?

seunlanlege · 2019-10-28T07:13:20Z

yeah sure

* add new line after writing block to hex file. * refactor for testability * correct import * better error reporting, code formatting * multiline imports * docs * better docs, move type to common types, merge ImportBlocks and ExportBlocks * tabs over spaces * correct test imports * Apply suggestions from code review Co-Authored-By: David <dvdplm@gmail.com> * correct typo * fixed test import

* Add new line after writing block to hex file. (#10984) * remove Drop impl

…m into v2.5.10-stable * 'v2.5.10-stable' of github.com:paritytech/parity-ethereum: [stable]: backport #10984 (#11212)

* ropsten #6631425 foundation #8798209 (#11201) * [stable] builtin, istanbul and mordor testnet backports (#11234) * ethcore-builtin (#10850) * [builtin]: support `multiple prices and activations` in chain spec (#11039) * [chain specs]: activate `Istanbul` on mainnet (#11228) * ethcore/res: add mordor testnet configuration (#11200) * Update list of bootnodes for xDai chain (#11236) * ethcore: remove `test-helper feat` from build (#11047) * Secret store: fix Instant::now() related race in net_keep_alive (#11155) (#11159) * [stable]: backport #10691 and #10683 (#11143) * Fix compiler warning (that will become an error) (#10683) * Refactor Clique stepping (#10691) * Add Constantinople eips to the dev (instant_seal) config (#10809) * Add cargo-remote dir to .gitignore (?) * Insert explicit warning into the panic hook (#11225) * Fix docker centos build (#11226) * Update MIX bootnodes. (#11203) * Use provided usd-per-eth value if an endpoint is specified (#11209) * Add new line after writing block to hex file. (#10984) * Type annotation for next_key() matching of json filter options (#11192) (but no `FilterOption` in 2.5 so…) * Upgrade jsonrpc to latest (#11206) * [CI] check evmbin build (#11096) * Correct EIP-712 encoding (#11092) * [client]: Fix for incorrectly dropped consensus messages (#11086) * Fix block detail updating (#11015) * Switching sccache from local to Redis (#10971) * Made ecrecover implementation trait public (#11188) * [dependencies]: jsonrpc `14.0.1` (#11183) * [receipt]: add `sender` & `receiver` to `RichReceipts` (#11179) * [ethcore/builtin]: do not panic in blake2pricer on short input (#11180) * util Host: fix a double Read Lock bug in fn Host::session_readable() (#11175) * ethcore client: fix a double Read Lock bug in fn Client::logs() (#11172) * Change how RPCs eth_call and eth_estimateGas handle "Pending" (#11127) * Cleanup stratum a bit (#11161) * Upgrade to jsonrpc v14 (#11151) * SecretStore: expose restore_key_public in HTTP API (#10241)

* [CI] check evmbin build (#11096) * Correct EIP-712 encoding (#11092) * [client]: Fix for incorrectly dropped consensus messages (#11082) (#11086) * Update hardcoded headers (foundation, classic, kovan, xdai, ewc, ...) (#11053) * Add cargo-remote dir to .gitignore (?) * Update light client headers: ropsten 6631425 foundation 8798209 (#11201) * Update list of bootnodes for xDai chain (#11236) * ethcore/res: add mordor testnet configuration (#11200) * [chain specs]: activate Istanbul on mainnet (#11228) * [builtin]: support multiple prices and activations in chain spec (#11039) * [receipt]: add sender & receiver to RichReceipts (#11179) * [ethcore/builtin]: do not panic in blake2pricer on short input (#11180) * Made ecrecover implementation trait public (#11188) * Fix docker centos build (#11226) * Update MIX bootnodes. (#11203) * Insert explicit warning into the panic hook (#11225) * Use provided usd-per-eth value if an endpoint is specified (#11209) * Cleanup stratum a bit (#11161) * Add Constantinople EIPs to the dev (instant_seal) config (#10809) (already backported) * util Host: fix a double Read Lock bug in fn Host::session_readable() (#11175) * ethcore client: fix a double Read Lock bug in fn Client::logs() (#11172) * Type annotation for next_key() matching of json filter options (#11192) * Upgrade jsonrpc to latest (#11206) * [dependencies]: jsonrpc 14.0.1 (#11183) * Upgrade to jsonrpc v14 (#11151) * Switching sccache from local to Redis (#10971) * Snapshot restoration overhaul (#11219) * Add new line after writing block to hex file. (#10984) * Pause pruning while snapshotting (#11178) * Change how RPCs eth_call and eth_estimateGas handle "Pending" (#11127) * Fix block detail updating (#11015) * Make InstantSeal Instant again #11186 * Filter out some bad ropsten warp snapshots (#11247)

seunlanlege added the M4-core ⛓ Core client code / Rust. label Aug 21, 2019

dvdplm reviewed Aug 21, 2019

View reviewed changes

seunlanlege force-pushed the fix-hex-block-import branch 2 times, most recently from 4b6353c to 761c851 Compare August 23, 2019 16:20

Demi-Marie previously requested changes Aug 24, 2019

View reviewed changes

niklasad1 reviewed Aug 26, 2019

View reviewed changes

ethcore/client-traits/src/lib.rs Outdated Show resolved Hide resolved

niklasad1 reviewed Aug 26, 2019

View reviewed changes

ethcore/src/client/client.rs Outdated Show resolved Hide resolved

niklasad1 reviewed Aug 26, 2019

View reviewed changes

ethcore/src/client/client.rs Outdated Show resolved Hide resolved

niklasad1 reviewed Aug 26, 2019

View reviewed changes

parity/blockchain.rs Outdated Show resolved Hide resolved

ordian approved these changes Aug 26, 2019

View reviewed changes

ordian added the A8-looksgood 🦄 Pull request is reviewed well. label Aug 26, 2019

ordian added this to the 2.7 milestone Aug 26, 2019

dvdplm suggested changes Aug 27, 2019

View reviewed changes

seunlanlege force-pushed the fix-hex-block-import branch 2 times, most recently from 9cfa4aa to fcc291f Compare September 4, 2019 10:40

dvdplm approved these changes Sep 5, 2019

View reviewed changes

ethcore/client-traits/src/lib.rs Outdated Show resolved Hide resolved

ethcore/client-traits/src/lib.rs Outdated Show resolved Hide resolved

ethcore/src/client/client.rs Outdated Show resolved Hide resolved

ethcore/src/client/client.rs Outdated Show resolved Hide resolved

dvdplm requested a review from Demi-Marie September 5, 2019 14:04

seunlanlege and others added 11 commits September 6, 2019 12:51

add new line after writing block to hex file.

ea8b547

refactor for testability

a4768be

correct import

5370bc2

better error reporting, code formatting

facd079

multiline imports

0dad5b2

docs

cabe3b2

better docs, move type to common types, merge ImportBlocks and Export…

2c812bd

…Blocks

tabs over spaces

92129e6

correct test imports

1bea879

Apply suggestions from code review

e1106a0

Co-Authored-By: David <dvdplm@gmail.com>

correct typo

bc19a0d

seunlanlege force-pushed the fix-hex-block-import branch from a44231a to bc19a0d Compare September 6, 2019 12:21

fixed test import

abc42f6

dvdplm merged commit 80f0e4b into master Sep 9, 2019

niklasad1 deleted the fix-hex-block-import branch September 9, 2019 12:08

dvdplm added B0-patch-stable 🕷 Pull request should also be back-ported to the stable branch. B1-patch-beta 🕷🕷 labels Oct 27, 2019

seunlanlege added a commit that referenced this pull request Oct 28, 2019

Add new line after writing block to hex file. (#10984)

d7d6ae0

seunlanlege mentioned this pull request Oct 28, 2019

[stable]: backport #10984 #11212

Merged

seunlanlege added a commit that referenced this pull request Oct 28, 2019

Add new line after writing block to hex file. (#10984)

ea7b318

This was referenced Nov 5, 2019

v2.5.10 stable #11239

Merged

v2.6.5 beta #11240

Merged

seunlanlege added a commit that referenced this pull request Nov 5, 2019

Add new line after writing block to hex file. (#10984)

7544dfb

seunlanlege added a commit that referenced this pull request Nov 8, 2019

Add new line after writing block to hex file. (#10984)

4e76ccc

seunlanlege pushed a commit that referenced this pull request Nov 8, 2019

[stable]: backport #10984 (#11212)

dd10c0d

* Add new line after writing block to hex file. (#10984) * remove Drop impl

dvdplm added a commit that referenced this pull request Nov 8, 2019

Merge branch 'v2.5.10-stable' of github.com:paritytech/parity-ethereu…

c1d0a8b

…m into v2.5.10-stable * 'v2.5.10-stable' of github.com:paritytech/parity-ethereum: [stable]: backport #10984 (#11212)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new line after writing block to hex file. #10984

Add new line after writing block to hex file. #10984

seunlanlege commented Aug 21, 2019 •

edited

Loading

dvdplm left a comment

Demi-Marie left a comment

ordian left a comment

dvdplm left a comment

dvdplm commented Sep 6, 2019

dvdplm commented Sep 7, 2019

dvdplm commented Oct 27, 2019

dvdplm commented Oct 27, 2019

seunlanlege commented Oct 28, 2019

Add new line after writing block to hex file. #10984

Add new line after writing block to hex file. #10984

Conversation

seunlanlege commented Aug 21, 2019 • edited Loading

dvdplm left a comment

Choose a reason for hiding this comment

Demi-Marie left a comment

Choose a reason for hiding this comment

ordian left a comment

Choose a reason for hiding this comment

dvdplm left a comment

Choose a reason for hiding this comment

dvdplm commented Sep 6, 2019

dvdplm commented Sep 7, 2019

dvdplm commented Oct 27, 2019

dvdplm commented Oct 27, 2019

seunlanlege commented Oct 28, 2019

seunlanlege commented Aug 21, 2019 •

edited

Loading