RethrowPolicy is not logging exceptions, unlike ErrorPolicy

[coot]

The following error is not logged to the log file, but only shows in stdout (probably because of GHC top level error handler):

cardano-node: ExceptionInLinkedThread "ThreadId 47" (ApiMisuse (ClosedDBError (Just (FsError {fsErrorType = FsDeviceFull, fsErrorPath = /mnt/HC_Volume_17016868/cardano-node/blocks/volatile/blocks-7685.dat, fsErrorString = "No space left on device", fsErrorNo = Just (Errno 28), fsErrorStack = CallStack (from HasCallStack):
  prettyCallStack, called at src/Ouroboros/Consensus/Storage/FS/API/Types.hs:297:23 in ouroboros-consensus-0.1.0.0-3e0c35ca6cdc55ae943ecbdd5636e68d9b1f1e912253d8a7ff6d1af1a76dd0d5:Ouroboros.Consensus.Storage.FS.API.Types
  ioToFsError, called at src/Ouroboros/Consensus/Storage/FS/IO.hs:88:41 in ouroboros-consensus-0.1.0.0-3e0c35ca6cdc55ae943ecbdd5636e68d9b1f1e912253d8a7ff6d1af1a76dd0d5:Ouroboros.Consensus.Storage.FS.IO
  handleError, called at src/Ouroboros/Consensus/Storage/FS/IO.hs:84:23 in ouroboros-consensus-0.1.0.0-3e0c35ca6cdc55ae943ecbdd5636e68d9b1f1e912253d8a7ff6d1af1a76dd0d5:Ouroboros.Consensus.Storage.FS.IO
  rethrowFsError, called at src/Ouroboros/Consensus/Storage/FS/IO.hs:54:39 in ouroboros-consensus-0.1.0.0-3e0c35ca6cdc55ae943ecbdd5636e68d9b1f1e912253d8a7ff6d1af1a76dd0d5:Ouroboros.Consensus.Storage.FS.IO
  hPutSome, called at src/Ouroboros/Consensus/Storage/FS/API.hs:103:5 in ouroboros-consensus-0.1.0.0-3e0c35ca6cdc55ae943ecbdd5636e68d9b1f1e912253d8a7ff6d1af1a76dd0d5:Ouroboros.Consensus.Storage.FS.API
  hPutAllStrict, called at src/Ouroboros/Consensus/Storage/FS/API.hs:280:19 in ouroboros-consensus-0.1.0.0-3e0c35ca6cdc55ae943ecbdd5636e68d9b1f1e912253d8a7ff6d1af1a76dd0d5:Ouroboros.Consensus.Storage.FS.API
  hPutAll, called at src/Ouroboros/Consensus/Storage/VolatileDB/Impl.hs:366:39 in ouroboros-consensus-0.1.0.0-3e0c35ca6cdc55ae943ecbdd5636e68d9b1f1e912253d8a7ff6d1af1a76dd0d5:Ouroboros.Consensus.Storage.VolatileDB.Impl, fsLimitation = False}))))

The node is configured with EnabledP2P: true.

RethrowPolicy are different in that matter to ErrorPolicies. The former don't log exceptions, while the latter do.

We can either:

• declare that RethrowPolicy should not log exceptions as this will duplicate error messages in many cases. For example exceptions thrown by any mini-protocol are logged by ConnectionHandlerTracer (which is part of ConnectionManagerTracer). Thanks to that runRethrowPolicy is a pure function.
• make runRethrowPolicy an effectuful function, which log exceptions.

The advantage of the latter approach is that:

• we don't miss errors like the above (which is thrown not from a connection thread), but somewhere from consensus code which is using RethrowPolicy (this is also easy to fix on the consensus side)
• we have one tracer which logs all important errors. This can be turned on by default (or not even configurable!).

[coot]

I looked more closely into this, it seems for me that the consensus needs to install a handler which logs exceptions thrown by this code.

On the networking side we can wrap the exceptions we rethrow in a wrapper, so they are not cached again by the consensus (e.g. some of the exception we handle come from consensus). This would make it much easier to reuse consensusRethrowPolicy.