track: Blockchain Lag

[cdecker]

When a node is offline, it may start to accumulate /blockchain
lag/. Blockchain lag is defined as the number of blocks that are in
the blockchain, but that the node has not yet seen and processed.

A large blockchain lag can cause a number of symptoms:

• Failing payments with WIRE_UNKNOWN_INVOICE_DETAILS / 16399
  despite the invoice definitely coming from the recipient: the CLTV
  is too low for the recipient to accept it, hence it sends back a
  signal that some choice on the sender was wrong. This can be a
  disagreement on payment rules, or in this case a disagreement on
  what is far enough in the future for the payment to be safe.
• Missing incoming on-chain payment: we will see them and add them
  once we process the block that contained the transaction.
• Missing confirmations for sent transactions, and missing spent
  marking of outputs used in a sent transaction: both of these will
  be set when we witness the transaction in a block. Before that you
  may refer to the reserved flag in listfunds to determine if an
  output is likely soon to be marked as spent and should therefore
  not be used. This reflects the intrinsic issue with confirmations
  on blockchains, since there is this limbo state between broadcast
  and confirmation, and so we chose to expose these details rather
  than pretending that broadcast is confirming.

For all of these the cause is the same, and addressing it,
automatically addresses them too. As such we are working on improving
the blockchain lag, by synchronizing in the background (not counted
towards the node's runtime for accounting!), and to optimize the sync
of the last couple of blocks for interactive (user) sessions.

[cdecker]

The blockchain lag issues have been resolved on the server side. With the inclusion of ElementsProject/lightning#7190, the problems related to small blockchain lag have also been mostly addressed. Closing.