Pause pruning while snapshotting

[dvdplm]

This PR pauses pruning while a snapshot is under way. It is very similar to previous attempts that was closed due to the problem with increased memory consumption. While this PR improves the situation slightly as it allows pruning to continue until the block we're snapshotting at, the objection is still valid: if this lands taking a snapshot will cause the memory limits set by the user to be ignored while a snapshot happens.

The default values for pruning-history and pruning-memory are 64 blocks and 32Mb respectively. At the time of writing, producing a snapshot on mainnet takes more than 10h on a fast machine and up to 25h on common hardware. This means that full non-archive nodes need to be configured to run with significantly higher values for --pruning-history or --pruning-memory. Empirically, values above 4000 blocks / 600Mb might work, but these numbers vary significantly depending on the hardware available and the load the machine is under otherwise.
There is obviously no incentive for a node operator to run such a configuration only to enable snapshots for new users.

The consequence is a dearth of recent snapshots available to new users trying to warp-sync, and a lot of wasted resources for full nodes attempting, and failing, to create snapshots.

Pausing pruning is bad. Keeping things as they are is also bad.

Here's a list of alternatives explored:

• Disable the node while snapshotting: no new blocks are imported, hence no pruning and the pruning memory stays stable. Turning off people's nodes for 20h seems like a bad idea.
• Don't create snapshots by default on non-archive nodes, i.e. make --no-periodic-snapshot the default. This will likely leave very few snapshot producing nodes available and lead to increased centralization. This is in a sense the current situation as no node running with a default config can produce snapshots.
• Make deep changes to the state trie to track the block number that last modified each node and teach the pruning algorithm to keep state that is needed for the current snapshot. This is non-trivial and even if attempted will not provide a timely resolution to the immediate issue of snapshot availability.
• Make a copy of the data needed for the snapshot into a new rocksdb database and make a snapshot of that instead. This would likely require a very large chunk of disk space be fairly involved.
• Abandon warp-sync and until an alternative is available users must slow-sync their nodes.

Measuring the impact of pausing pruning is not straight forward. The time to take a snapshot varies significantly in between runs and from host to host. The pruneable set, frozen in this PR, increases RAM usage between 500 - 1800Mb, e.g.:

// journal mem usage: 1 852Mb (at this point my node lost all peers and stopped importing blocks, which stops pruning attempts)
2019-10-19 11:26:19  Verifier #7 TRACE pruning  Pruning is freezed at era 8767200; earliest era=8767200, latest era=8770464, journal_size=1966523469, mem_used=1942266864. Not pruning.
…
2019-10-19 16:15:42  Periodic Snapshot DEBUG snapshot  Took a snapshot of 71000605 accounts
2019-10-19 16:15:42  Periodic Snapshot INFO snapshot  produced 3638 state chunks and 97 block chunks.
2019-10-19 16:15:42  Periodic Snapshot TRACE snapshot  (defer) Unfreezing pruning, setting snapshotting_at back to 0
2019-10-19 16:15:42  Periodic Snapshot INFO snapshot::service  Finished taking snapshot at #8767200, in 58180.542310568s

The important code change here is this, the rest is mostly cosmetics.

[rphmeier]

with this many fields, it's almost certainly better to use a single parking lot mutex wrapping a single struct containing all this data. maybe beyond the scope

[dvdplm]

Good idea, I'll make the change in a diff PR as it gets a bit noisy.

[arkpar]

Make a copy of the data needed for the snapshot into a new rocksdb database and make a snapshot of that instead. This would likely require a very large chunk of disk space be fairly involved.

That was one of the goals for parity-db.
The canonical state stored not in the trie, but directly in the database will take a fraction of the space. It will also help greatly with query performance.

[dvdplm]

@rphmeier @arkpar thank you for your input. I'll collect a few more opinions but for the time being I think pausing pruning is the least worst choice we have for the short term.
Hopefully it'll give us enough time to plan the next steps, parity-db or other.

[ngotchac]

LGTM other than a few questions.