Discovery: throttle node DB commits (#3581)

[battlmonstr]

UpdateFindFails/UpdateLastPingReceived/UpdateLastPongReceived events
are causing bursty DB commits (100 per minute).

This optimization throttles the commits to happen at most once in a few seconds,
because this info doesn't need to be persisted immediately.

[AskAlexSharov]

Please, Create one more PR from stable branch with cherry-pick of this pr. Thank you

[battlmonstr]

👌 some tests fail, I'm gonna check them first.

[battlmonstr]

The current solution doesn't work well, because there can be only one pending RW transaction globally. Having a separate open transaction for pings makes other writes impossible. For example, updating the local node info blocks forever.

I'd either need to channel all node DB access to this event loop, or make an alternative solution. I'm considering trying something like MDBX_SAFE_NOSYNC instead.

[AskAlexSharov]

@battlmonstr MDBX_SAFE_NOSYNC doesn't conceptually change anything - still can't open parallel write txs.

FYI, TxPool works next way:

• updates are happening in RAM (in btree, aka cache)
• 1 goroutine with infinity loop periodically takes lock on btree and flushing it to db and cleaning btree. https://github.com/ledgerwatch/erigon-lib/blob/a2cbecf3730e369022806f5ae451c710826b4473/txpool/pool.go#L1333
• read requests - do read from cache with fallback to read tx on db.

Can you explain one more time - what is current problem with having cache in nodedb? (kick me in Discord if you need)
"I'd either need to channel all node DB access to this event loop" - see also bbold.Batch

[AskAlexSharov]

rejecting until some discussion

[battlmonstr]

@AskAlexSharov the parallel writes are not needed anymore, I have removed that. It was a problem for the previous solution attempt with a goroutine, but I think that SafeNoSync alone solves the issue by and large.

The issue was reported in discord a while ago that the initial node sync performance is not great when using a HDD.
Alexey did some testing and felt that one of the reason was frequent commits to the nodedb.
Although it is 2 different database files, his hypothesis was that the sequential writes to the chain db are affected by the random writes to the nodedb. At the very least, it was unfortunate that nodedb did so many fsync commits while the info there is not needed to be 100% durable.

SafeNoSync

AFAIK, mdbx commit has 2 steps:

1. write data: after this the data is accepted by the OS, and will be written to disk eventually
2. flush OS buffers to disk (let's call it fsync): after this the data is fully persisted.

By default, mdbx tries to be 100% durable, and each Commit() does both write and fsync.
nodedb uses db.Update() calls, which are causing a Commit().
After this change, Update() calls cause fsync at most once in 2 seconds. mdbx has an internal timestamp of the last fsync, and it checks if the time since then is greater than the SyncPeriod.

P.S. If there are no new updates for a long time we risk to lose 2 seconds worth of updates. Here this doesn't seem like a big deal.

[AskAlexSharov]

agree with everything