feat: chainstore: batch writes of tipsets #10800
Conversation
chain/store/store.go
Outdated
| return xerrors.Errorf("failed to persist block headers: %w", err) | ||
| func (cs *ChainStore) PersistTipsets(ctx context.Context, tipsets []*types.TipSet) error { | ||
| toPersist := make([]*types.BlockHeader, 0, len(tipsets)*int(build.BlocksPerEpoch)) | ||
| tsBlks := make([]block.Block, 0, len(tipsets)*int(build.BlocksPerEpoch)) |
There was a problem hiding this comment.
| tsBlks := make([]block.Block, 0, len(tipsets)*int(build.BlocksPerEpoch)) | |
| tsBlks := make([]block.Block, 0, len(tipsets)) |
chain/store/snapshot.go
Outdated
| @@ -123,11 +123,9 @@ func (cs *ChainStore) Import(ctx context.Context, r io.Reader) (*types.TipSet, e | |||
| } | |||
|
|
|||
| ts := root | |||
| tssToPersist := make([]*types.TipSet, 0, uint64(TipsetkeyBackfillRange)*build.BlocksPerEpoch) | |||
There was a problem hiding this comment.
| tssToPersist := make([]*types.TipSet, 0, uint64(TipsetkeyBackfillRange)*build.BlocksPerEpoch) | |
| tssToPersist := make([]*types.TipSet, 0, uint64(TipsetkeyBackfillRange)) |
chain/store/store.go
Outdated
| return xerrors.Errorf("failed to persist block headers: %w", err) | ||
| func (cs *ChainStore) PersistTipsets(ctx context.Context, tipsets []*types.TipSet) error { | ||
| toPersist := make([]*types.BlockHeader, 0, len(tipsets)*int(build.BlocksPerEpoch)) | ||
| tsBlks := make([]block.Block, 0, len(tipsets)*int(build.BlocksPerEpoch)) |
There was a problem hiding this comment.
nit: capacity should just be len(tipsets) since theres one ts key per tipset. You could also just use the iteration index from range tipsets and set the slice directly since you know the exact size on initialization unlike the slice of blocks.
chain/store/snapshot.go
Outdated
| @@ -123,11 +123,9 @@ func (cs *ChainStore) Import(ctx context.Context, r io.Reader) (*types.TipSet, e | |||
| } | |||
|
|
|||
| ts := root | |||
| tssToPersist := make([]*types.TipSet, 0, uint64(TipsetkeyBackfillRange)*build.BlocksPerEpoch) | |||
There was a problem hiding this comment.
nit: drop *build.BlocksPerEpoch since one slice item per tipset
| if err != nil { | ||
| return nil, err | ||
| } | ||
| tssToPersist = append(tssToPersist, ts) | ||
| parentTsKey := ts.Parents() | ||
| ts, err = cs.LoadTipSet(ctx, parentTsKey) |
There was a problem hiding this comment.
Sanity checking since I don't know this code: is this actually going to work now that we are not persisting tipsets one by one in this loop before the check? We're putting these tipsets to cs somewhere else first right?
There was a problem hiding this comment.
My understanding is yes, because we've already Put all of these objects into our store above. This loop is about:
- sanity checking that we have them
- writing the TsKeys into the store for Eth stuff
It's probably not actually any faster to batch this, but I figured we might as well for cleanness.
arajasek
force-pushed
the
asr/speedup-persist
branch
from
fd2459e to
49f5825
Compare
Related Issues
Helps with #10788, but does NOT close it.
Proposed Changes
Additional Info
When syncing calibnet, this reduces the time to persist all headers from 4-5 minutes, to < 15 seconds. This will have less of an impact for folks syncing from snapshots, but is still a worthwhile perf gain.
Checklist
Before you mark the PR ready for review, please make sure that:
<PR type>: <area>: <change being made>fix: mempool: Introduce a cache for valid signaturesPR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, testarea, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps