Implement the GDD governor

[facundominguez]

This PR builds upon #979 to provide the GDD governor, the component responsible for identifying and disconnecting peers offering sparser chains than the best. This has the effect of unblocking the Limit on Eagerness, since removing disagreeing peers allows the current selection to advance.

The GDD governor runs in a background thread and evaluates candidate chains whenever they change, or whenever a peer claims to have no more headers, or whenever a peer starts sending headers beyond the forecast horizon.

Whenever GDD disconnects peers, the chain selection is updated.

The implementation of the GDD governor is in commit 3fc6d28. A range of supporting changes is provided as well.

---

Peer simulator improvements in f02d8df include:

• Our simulator of peers now kills all miniprotocol threads whenever one of the miniprotocols is terminated. The GDD governor might terminate a ChainSync client, and this termination should also cause the termination of the BlockFetch client.
• Our tests now also check that no threads are left lingering when the peer simulator is terminated.
• Traces now print a timestamp when events occur between the start of consecutive ticks. This need arose when Limit on Patience started disconnecting peers because of exhausted buckets, and these events could happen at any moment between ticks. See this comment of Introduce a time-aware tracer #990 for an example trace.

---

Post-GDD improvements in ea12e12 include:

• Started tracing the information about BlockFetch timeouts and BlockFetch logic decisions. This information became relevant when we had to analyze why blocks wouldn't be downloaded from the honest peer, or why a quiet adversarial peer wasn't being disconnected.
• Have the Limit on Eagerness recalculated when the change selection changes. Limit on Eagerness does not depend on the chain selection, but this mechanism has the desired side effect of retriggering chain selection in succession until no more blocks are added to it. This change is necessary after chain selection stopped being idempotent in 7ace045 (later on this same PR). This change is included here to demonstrate that the GDD governor prevents the node from being leashed, but otherwise we intend to recover idempotency of chain selection in a later PR.

---

Tests for the GDD in b8af4b4 include:

• Added a new test to check that GDD actually disconnects nodes in small examples. This test is subsumed by the already existing test "serve adversarial branches", but it tries only small peer schedules which are easier to approach on test failures.
• Enabled one of the leashing attack tests. This test was waiting for the GDD governor to be implemented. It generates an honest peer schedule for every peer, and then drops from the adversary schedules some ticks chosen at random. This has the effect of having the adversaries introduce delays to send headers or blocks that shouldn't stop the node from reaching the honest tip by the end of the test.

There is one other leashing attack test called "time limited leashing attack", which it is enabled in a follow up PR.

---

The classifiers added in cfefa6c keep track of whether peers are being disconnected by the GDD governor, and the LoP, and timeouts. It also keep tracks of adversaries who did a rollback of their chain. The commit also implements a check ensuring no peer is killed because of an unforeseen exception.

---

Chain generators implement the extended praos chain growth assumption (EPCG) in d8c6546. This causes the honest chain to have at least k+1 blocks in every stability window. The former Praos Chain Growth assumption required at least k, and in practice chains with less than k+1 blocks in a stability window are still very rare. The EPCG assumption makes a difference to testing the GDD governor.

We cannot let the almost caught up GDD governor disconnect honest peers which disagree on a few final blocks. Therefore, we ask the governor to require a disagreement of more than k blocks before deciding a disconnection. But asking for more than k blocks without the EPCG assumption means that the GDD governor might not be able to make progress if the k+1st block is after the first stability window after the intersection and after the forecast horizon. In this case, the candidate fragments wouldn't be extended with the k+1st block, and the GDD governor wouldn't realize that there is a discrepancy of more than k blocks.

---

In c8468c2 we don't allow the chain selection to fork more than k blocks from the honest chain. The honest chain is considered to be the one containing the first intersection at which candidate fragments fork into different chains.

It used to be the case that we allowed the selection to choose up to k blocks after the intersection. But it turned out that if the selection tip is still X blocks behind the intersection, allowing chain selection to pick X+k blocks, there is no guarantee that the selection will only add blocks from the candidate fragments. It could select more than k blocks of an older fork in the chain DB, and this is catastrophic because now the immutable tip would leave the honest chain.

---

bf3f09a refactors and properly bundles the various state that a ChainSync client maintains.

[dnadales]

Minor, this comment might get outdated if shortWait changes. Would it make sense to use something like Just _10Seconds or just the numeric literal. Alternatively, could shortWait be a local definition?

[amesgen]

Good point, I instead removed the comment to avoid it going out of sync. shortWait is used in various places, so knowing its value is already necessary in various places.

[dnadales]

Would it make sense to guard against the case in which peer is not in the map?

[amesgen]

This is actually part of the QSM precondition 👍

[dnadales]

We should either use past tense here, or present in the other fragments :)

[amesgen]

I think "Set ..." is conveniently both present and past tense 😅