Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync master with devel #1673

Merged
merged 70 commits into from
Sep 16, 2020
Merged
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
ec61086
update Grafana dashboard
stefantalpalaru Aug 23, 2020
a7a279d
add option to disable discv5 (#1509)
arnetheduck Aug 24, 2020
0edacfe
important book updates (#1561)
unixpi Aug 24, 2020
ddd8489
bump vendor/nim-blscurve (#1560)
stefantalpalaru Aug 24, 2020
ab22bad
bump vendor/nim-metrics (#1565)
stefantalpalaru Aug 25, 2020
f26d6a4
reuse validator key cache better (#1562)
arnetheduck Aug 26, 2020
81788be
Fork choice - almost free pruning - fix #1534 (#1535)
mratsim Aug 26, 2020
b525dc9
Request manager ignores non-critical errors while processing blocks. …
cheatfate Aug 26, 2020
83667dd
harden beacon_pending_deposits metrics calculation against overflow (…
tersec Aug 26, 2020
142f791
Jenkins: run benchmarks (#1524)
stefantalpalaru Aug 26, 2020
950da7a
Medalla guide updates + troubleshooting page (#1571)
unixpi Aug 26, 2020
2073977
Update the Nimbus bootstrap nodes
zah Aug 26, 2020
2081c4f
avoid unnecessary seq allocations (#1573)
arnetheduck Aug 27, 2020
fa1621d
implement clock disparity for attestation validation (#1568)
arnetheduck Aug 27, 2020
c810b64
log getblocks error
arnetheduck Aug 27, 2020
b8da265
add setting for benchmarking and profiling of sqlite block storage ti…
tersec Aug 27, 2020
ecc70c6
remove BUILD_LOG_LEVEL
stefantalpalaru Aug 27, 2020
504e96c
fix medalla-dev
stefantalpalaru Aug 27, 2020
b457da3
update troubleshooting medalla (#1578)
unixpi Aug 27, 2020
f250012
Makefile: add clean-cross task for cross compiling (#1554)
zedeus Aug 28, 2020
d9f926b
bump nim-eth (#1579)
tersec Aug 31, 2020
ab25566
bound block quarantine size (#1564)
tersec Aug 31, 2020
02ddc41
ignore sqlite WAL journals in git; increase logging priority of attes…
tersec Aug 31, 2020
124ec4b
perform slashing check before mutation (#1594)
arnetheduck Sep 1, 2020
43c6a36
remove template expansion file from Git control (#1589)
stefantalpalaru Sep 1, 2020
67d73c4
added the --network=<x> option to the tools for which it matters
onqtam Sep 1, 2020
6000a49
auto-bump nim-libp2p (#1484)
nbc-bump-bot[bot] Sep 1, 2020
65d7787
50/50 bn/vc split for the validator keys ON by default for the testne…
onqtam Sep 1, 2020
d9f9949
use a separate process for the private keys (Off by default) - there …
onqtam Sep 1, 2020
e4a43f7
address issue #1580 (#1600)
tersec Sep 2, 2020
aa3298f
support spaces in $(MAKE) (#1603)
stefantalpalaru Sep 4, 2020
aed57df
avoid hash tree root calculation when loading blocks from database (#…
arnetheduck Sep 4, 2020
456bdc8
address issue #1552 (#1601)
tersec Sep 4, 2020
f61fdf9
bump chronos (#1598)
arnetheduck Sep 4, 2020
c5f8ef8
auto-bump nim-libp2p (#1604)
nbc-bump-bot[bot] Sep 5, 2020
d584591
simplify libp2p logging (#1605)
arnetheduck Sep 6, 2020
4c3dd85
bump libp2p (#1609)
arnetheduck Sep 7, 2020
3d5f24f
stop discarding future epochs; remove a StateCache() construction (#1…
tersec Sep 7, 2020
dde26f3
better state cache reuse (#1612)
arnetheduck Sep 8, 2020
b3b5785
mark comments documenting functions as a whole as such (#1613)
tersec Sep 8, 2020
d0de1a4
Fix some warnings and hints and partly revert #1610 (#1615)
tersec Sep 8, 2020
990bc2e
remove obsolete comment
tersec Sep 9, 2020
5d1ea5e
match log filenames more precisely for logtrace analysis (#1617)
tersec Sep 9, 2020
67f44d4
use V=1 in Jenkinsfile (#1618)
tersec Sep 9, 2020
7c21fa9
bump libp2p, introduce AsyncQueue BufferStream (#1599)
arnetheduck Sep 10, 2020
697bd23
fix low peer count command typo (#1619)
unixpi Sep 10, 2020
a087909
fix req/resp protocol (#1621)
arnetheduck Sep 10, 2020
9db05c4
Bump chronos to help investigate Index problem. (#1625)
cheatfate Sep 11, 2020
8a5a261
Quick fix to prune some states, pending smarter state storage (#1624)
arnetheduck Sep 11, 2020
775683c
await validator duties (#1626)
arnetheduck Sep 11, 2020
c7c9b9d
Syncing V2 (#1602)
cheatfate Sep 11, 2020
48893f1
add ncli_db subcommand to prune database of unnecessary blocks and st…
tersec Sep 11, 2020
eaea3db
remove chronos future tracking from default build (#1628)
arnetheduck Sep 11, 2020
5fff800
Bump nim-eth to use lrucache for discovery sessions (#1622)
kdeme Sep 11, 2020
aca1a31
cleanly close kvstore databases and bump nim-eth (#1630)
tersec Sep 12, 2020
e6c82d1
Update medalla-troubleshooting.md (#1633)
unixpi Sep 13, 2020
7d00ca2
Update medalla-troubleshooting.md (#1635)
unixpi Sep 13, 2020
ca2e7e2
update medalla guide: advanced options (WIP) (#1623)
unixpi Sep 13, 2020
60b8905
reorder startup steps (#1642)
stefantalpalaru Sep 14, 2020
9f21bbd
[WIP] skeleton of attester slashing pool & validators (#1639)
tersec Sep 14, 2020
c76305f
fix some todo (#1645)
arnetheduck Sep 14, 2020
617c295
Bump chronos to fix memory leaks. (#1648)
cheatfate Sep 15, 2020
7e10b98
enable topic unsubscribing and attestation subnet cycling (#1646)
tersec Sep 15, 2020
88eb0c0
auto-bump nim-libp2p (#1606)
nbc-bump-bot[bot] Sep 15, 2020
9abdbda
Fix sync_manager.nim not rewarding peers for good responses. (#1660)
cheatfate Sep 16, 2020
6d8130d
close block_sim database; remove code duplication in exit_pool (#1656)
tersec Sep 16, 2020
6e46325
PeerPool fixes. (#1654)
cheatfate Sep 16, 2020
52548f0
Opt-in Slashing protection + interchange (#1643)
mratsim Sep 16, 2020
df685f0
auto-bump nim-libp2p (#1659)
nbc-bump-bot[bot] Sep 16, 2020
ce85f58
better handling of Ctrl+C during `make update` (#1588)
stefantalpalaru Sep 16, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add ncli_db subcommand to prune database of unnecessary blocks and st…
…ates (#1593)

* add ncli_db subcommand to prune database of unnecessary blocks, states, and state roots

* tweak comments

* reduce default aggressiveness in pruning old states

* move copyPrunedDatabase() to ncli_db, as it's not generally useful as part of beacon_chain_db and doesn't use any internal interfaces
tersec authored Sep 11, 2020

Verified

This commit was signed with the committer’s verified signature.
SuperQ Ben Kochie
commit 48893f1c2ed5eb9c77591aa26ea8789cd474470f
4 changes: 1 addition & 3 deletions beacon_chain/block_pools/block_pools_types.nim
Original file line number Diff line number Diff line change
@@ -177,9 +177,7 @@ type
data*: HashedBeaconState

blck*: BlockRef ##\
## The block associated with the state found in data - normally
## `blck.state_root == data.root` but the state might have been advanced
## further with empty slots invalidating this condition.
## The block associated with the state found in data

BlockSlot* = object
## Unique identifier for a particular fork and time in the block chain -
93 changes: 93 additions & 0 deletions ncli/ncli_db.nim
Original file line number Diff line number Diff line change
@@ -22,6 +22,7 @@ type
bench
dumpState
dumpBlock
pruneDatabase
rewindState

# TODO:
@@ -59,6 +60,17 @@ type
argument
desc: "Block roots to save".}: seq[string]

of pruneDatabase:
dryRun* {.
defaultValue: false
desc: "Don't write to the database copy; only simulate actions; default false".}: bool
keepOldStates* {.
defaultValue: true
desc: "Keep pre-finalization states; default true".}: bool
verbose* {.
defaultValue: false
desc: "Enables verbose output; default false".}: bool

of rewindState:
blockRoot* {.
argument
@@ -157,6 +169,85 @@ proc cmdDumpBlock(conf: DbConf) =
except CatchableError as e:
echo "Couldn't load ", blockRoot, ": ", e.msg

proc copyPrunedDatabase(
db: BeaconChainDB, copyDb: BeaconChainDB,
dryRun, verbose, keepOldStates: bool) =
## Create a pruned copy of the beacon chain database

let
headBlock = db.getHeadBlock()
tailBlock = db.getTailBlock()

doAssert headBlock.isOk and tailBlock.isOk
doAssert db.getBlock(headBlock.get).isOk
doAssert db.getBlock(tailBlock.get).isOk

var
beaconState: ref BeaconState
finalizedEpoch: Epoch # default value of 0 is conservative/safe
prevBlockSlot = db.getBlock(db.getHeadBlock().get).get.message.slot

beaconState = new BeaconState
let headEpoch = db.getBlock(headBlock.get).get.message.slot.epoch

# Tail states are specially addressed; no stateroot intermediary
if not db.getState(
db.getBlock(tailBlock.get).get.message.state_root, beaconState[],
noRollback):
doAssert false, "could not load tail state"
if not dry_run:
copyDb.putState(beaconState[])

for signedBlock in getAncestors(db, headBlock.get):
if not dry_run:
copyDb.putBlock(signedBlock)
if verbose:
echo "copied block at slot ", signedBlock.message.slot

for slot in countdown(prevBlockSlot, signedBlock.message.slot + 1):
if slot mod SLOTS_PER_EPOCH != 0 or
((not keepOldStates) and slot.epoch < finalizedEpoch):
continue

# Could also only copy these states, head and finalized, plus tail state
let stateRequired = slot.epoch in [finalizedEpoch, headEpoch]

let sr = db.getStateRoot(signedBlock.root, slot)
if sr.isErr:
if stateRequired:
doAssert false, "state root and state required"
continue

if not db.getState(sr.get, beaconState[], noRollback):
# Don't copy dangling stateroot pointers
if stateRequired:
doAssert false, "state root and state required"
continue

finalizedEpoch = max(
finalizedEpoch, beaconState.finalized_checkpoint.epoch)

if not dry_run:
copyDb.putStateRoot(signedBlock.root, slot, sr.get)
copyDb.putState(beaconState[])
if verbose:
echo "copied state at slot ", slot, " from block at ", shortLog(signedBlock.message.slot)

prevBlockSlot = signedBlock.message.slot

if not dry_run:
copyDb.putHeadBlock(headBlock.get)
copyDb.putTailBlock(tailBlock.get)

proc cmdPrune(conf: DbConf) =
let
db = BeaconChainDB.init(
kvStore SqStoreRef.init(conf.databaseDir.string, "nbc").tryGet())
copyDb = BeaconChainDB.init(
kvStore SqStoreRef.init(conf.databaseDir.string, "nbc_pruned").tryGet())

db.copyPrunedDatabase(copyDb, conf.dryRun, conf.verbose, conf.keepOldStates)

proc cmdRewindState(conf: DbConf, runtimePreset: RuntimePreset) =
echo "Opening database..."
let
@@ -191,5 +282,7 @@ when isMainModule:
cmdDumpState(conf)
of dumpBlock:
cmdDumpBlock(conf)
of pruneDatabase:
cmdPrune(conf)
of rewindState:
cmdRewindState(conf, runtimePreset)