From 72631213266c03b3ff3e8eb9590e47ff9271db39 Mon Sep 17 00:00:00 2001 From: Arnaud Bailly Date: Fri, 8 Sep 2023 17:09:57 +0200 Subject: [PATCH 01/14] Add ADR about current design of network resilience layer --- docs/adr/2023-09-08_026-network-resilience.md | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 docs/adr/2023-09-08_026-network-resilience.md diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md new file mode 100644 index 00000000000..6a7d0ad6092 --- /dev/null +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -0,0 +1,62 @@ +--- +slug: 26 +title: | + 26. Network failures model +authors: [] +tags: [Draft] +--- + +## Status + +Draft + +## Context + +The current Head cluster is very fragile as has been observed on several occasions: A single hiccup in the connectivity between nodes while a head is open and nodes are exchanging messages can very easily lead to the Head being stuck and require an emergency closing, possibly even manually. + +We want Hydra to be _Consistent_ in the presence of _Network Partitions_, under the _fail-recovery_ model assumption, eg. processes may fail by stopping and later recovering. Our system lies in the [CP](https://en.wikipedia.org/wiki/CAP_theorem) space of the landscape mapped by the CAP theorem. + +We have identified 3 main sources of failures in the _fail-recovery_ model that can lead to a head being stuck: + +1. The network layer can drop messages from the moment a node `broadcast`s it, leading to some messages not being received at the other end +2. The sending node can crash in between the moment the state is changed (and persisted) and the moment a message is actually sent through the network (or even it calls `broadcast`) +3. The receiving node can crash in between the moment the message has been received in the network layer, and it's processed (goes through the queue + +We agree that we'll want to address all those issues in order to provide a good user experience, as not addressing 2. and 3. can lead to hard to troubleshoot issues with heads. We have not experienced those issues yet as they would probably only crop up under heavy loads, or in the wild. But we also agree we want to tackle 1. first because it's where most of the risk lies. By providing a _Reliable Broadcast_ layer, we will significantly reduce the risks and can then later on address the other points. + +Therefore, the scope of this ADR is to address only point 1. above: Ensure broadcast messages are eventually received by all peers, or some timeout is reached. + +### Discussion + +* We are currently using the [ouroboros-framework](https://github.com/input-output-hk/ouroboros-network) and [typed-protocols](https://github.com/input-output-hk/typed-protocols) network stack as a mere [transport](https://osi-model.com/transport-layer/) layer. + * Being built on top of TCP, ourboros multiplexer (Mux) provides the same reliability guarantees, plus the multiplexing capabilities of course + * It also takes care of reconnecting to peers when a failure is detected which relieves us of doing so, but any reconnection implies a reset of each peer's state machine which means we need to make sure any change to the state of pending/received messages is handled by the applicative layer + * Our [FireForget](https://github.com/input-output-hk/hydra/blob/8a8e0829964132bde8949e5249a1ab303af92fb8/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L31) protocol is lame + * Ouroboros/typed-protocols provides enough machinery to implement a reliable broadcast protocol, for example by reusing existing `KeepAlive` protocol and building a more robust point-to-point protocol than what we have now + * There is a minor limitation, namely that the subscription mechanism does not handle connections invidually, but as a set of equivalent point-to-point full duplex connections whose size (valency) needs to be maintained at a certain threshold, which means that unless backed in the protocol itself, protocol state-machine and applications are not aware of the identity of the remote peer +* We have built our `Network` infrastructure over the concept of relatively independent layers, each implementing a similar interface with different kind of messages, to `broadcast` messages to all peers and be notified of incoming messages through a `callback`. + * This pipes-like abstraction allows us to compose our network stack like: + + ``` + withAuthentication (contramap Authentication tracer) signingKey otherParties $ + withHeartbeat nodeId connectionMessages $ + withOuroborosNetwork (contramap Network tracer) localhost peers + ``` + * This has the nice property that we can basically swap the lower layers should we need to, for example to use [UDP](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/UDP.hs), or add other layers for example to identify [Multiple Heads](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/MultiHead.hs#L26) + +## Decision + +* We implement our own message tracking and resending logic as a standalone `Network` layer +* That layer consumes and produces `Authenticate`d messages in order to identify the source of messages +* It uses a vector of monotonically increasing _sequence numbers_ associated with each party (including itself) to track what are the last messages from each party and to ensure FIFO delivery of messages +* This _vector_ is also used to identify peers which are lagging behind and to resend the missing messages, or to drop messages which have already been received + +## Consequences + +* We keep our existing `Network` interface hence all message will be resent to all peers + * This could be later optimised either by providing a smarter interface with a `send :: Peer -> msg -> m ()` unicast function, or by adding a layer with filtering capabilities, or both +* We need to ensure messages are not kept forever when resending, eg. that the pending messages list is garbage collected +* We need to refactor our `Heartbeat` layer to decouple the 2 capabilities it embeds, namely sending periodical heartbeats to peers, and providing listeners with information about a peer connection status, so that the `Reliability` layer can actually rely on those heartbeats to get regular update of peers' knowledge even when the actual Head is idle. +* We want to specify this protocol clearly in order to ease implementation in other languages, detailing the structure of messages and the semantics of retries and timeouts. +* We do not implement a _pull-based_ message communication mechanism as initially envisioned +* We do not persist messages either on the receiving or sending side From feed707bada057bd2dcfc2006059545fbd8d6a0f Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 11 Sep 2023 17:05:38 +0200 Subject: [PATCH 02/14] Minor fixes/clarifications --- docs/adr/2023-09-08_026-network-resilience.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 6a7d0ad6092..0745b25da11 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -20,7 +20,7 @@ We have identified 3 main sources of failures in the _fail-recovery_ model that 1. The network layer can drop messages from the moment a node `broadcast`s it, leading to some messages not being received at the other end 2. The sending node can crash in between the moment the state is changed (and persisted) and the moment a message is actually sent through the network (or even it calls `broadcast`) -3. The receiving node can crash in between the moment the message has been received in the network layer, and it's processed (goes through the queue +3. The receiving node can crash in between the moment the message has been received in the network layer, and it's processed (goes through the queue) We agree that we'll want to address all those issues in order to provide a good user experience, as not addressing 2. and 3. can lead to hard to troubleshoot issues with heads. We have not experienced those issues yet as they would probably only crop up under heavy loads, or in the wild. But we also agree we want to tackle 1. first because it's where most of the risk lies. By providing a _Reliable Broadcast_ layer, we will significantly reduce the risks and can then later on address the other points. @@ -31,7 +31,7 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * We are currently using the [ouroboros-framework](https://github.com/input-output-hk/ouroboros-network) and [typed-protocols](https://github.com/input-output-hk/typed-protocols) network stack as a mere [transport](https://osi-model.com/transport-layer/) layer. * Being built on top of TCP, ourboros multiplexer (Mux) provides the same reliability guarantees, plus the multiplexing capabilities of course * It also takes care of reconnecting to peers when a failure is detected which relieves us of doing so, but any reconnection implies a reset of each peer's state machine which means we need to make sure any change to the state of pending/received messages is handled by the applicative layer - * Our [FireForget](https://github.com/input-output-hk/hydra/blob/8a8e0829964132bde8949e5249a1ab303af92fb8/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L31) protocol is lame + * Our [FireForget](https://github.com/input-output-hk/hydra/blob/8a8e0829964132bde8949e5249a1ab303af92fb8/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L31) ignores connections/disconnections * Ouroboros/typed-protocols provides enough machinery to implement a reliable broadcast protocol, for example by reusing existing `KeepAlive` protocol and building a more robust point-to-point protocol than what we have now * There is a minor limitation, namely that the subscription mechanism does not handle connections invidually, but as a set of equivalent point-to-point full duplex connections whose size (valency) needs to be maintained at a certain threshold, which means that unless backed in the protocol itself, protocol state-machine and applications are not aware of the identity of the remote peer * We have built our `Network` infrastructure over the concept of relatively independent layers, each implementing a similar interface with different kind of messages, to `broadcast` messages to all peers and be notified of incoming messages through a `callback`. @@ -42,6 +42,7 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad withHeartbeat nodeId connectionMessages $ withOuroborosNetwork (contramap Network tracer) localhost peers ``` + * This has the nice property that we can basically swap the lower layers should we need to, for example to use [UDP](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/UDP.hs), or add other layers for example to identify [Multiple Heads](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/MultiHead.hs#L26) ## Decision @@ -54,9 +55,10 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad ## Consequences * We keep our existing `Network` interface hence all message will be resent to all peers - * This could be later optimised either by providing a smarter interface with a `send :: Peer -> msg -> m ()` unicast function, or by adding a layer with filtering capabilities, or both + * This could be later optimized either by providing a smarter interface with a `send :: Peer -> msg -> m ()` unicast function, or by adding a layer with filtering capabilities, or both * We need to ensure messages are not kept forever when resending, eg. that the pending messages list is garbage collected * We need to refactor our `Heartbeat` layer to decouple the 2 capabilities it embeds, namely sending periodical heartbeats to peers, and providing listeners with information about a peer connection status, so that the `Reliability` layer can actually rely on those heartbeats to get regular update of peers' knowledge even when the actual Head is idle. * We want to specify this protocol clearly in order to ease implementation in other languages, detailing the structure of messages and the semantics of retries and timeouts. * We do not implement a _pull-based_ message communication mechanism as initially envisioned -* We do not persist messages either on the receiving or sending side +* We do not persist messages either on the receiving or sending side at this time +* We may consider relying on the vector clock in the future to ensure perfect ordering of messages on each peer and make impossible for legit transactions to be temporarily seen as invalid. This can happen in the current version and is handled through wait and ttl From 81c6daea7da43831769007d0860d83eef8572bea Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Tue, 19 Sep 2023 16:33:13 +0200 Subject: [PATCH 03/14] Amend the ping process --- docs/adr/2023-09-08_026-network-resilience.md | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 0745b25da11..387000c2d99 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -3,12 +3,12 @@ slug: 26 title: | 26. Network failures model authors: [] -tags: [Draft] +tags: [Proposed] --- ## Status -Draft +Proposed ## Context @@ -24,13 +24,13 @@ We have identified 3 main sources of failures in the _fail-recovery_ model that We agree that we'll want to address all those issues in order to provide a good user experience, as not addressing 2. and 3. can lead to hard to troubleshoot issues with heads. We have not experienced those issues yet as they would probably only crop up under heavy loads, or in the wild. But we also agree we want to tackle 1. first because it's where most of the risk lies. By providing a _Reliable Broadcast_ layer, we will significantly reduce the risks and can then later on address the other points. -Therefore, the scope of this ADR is to address only point 1. above: Ensure broadcast messages are eventually received by all peers, or some timeout is reached. +Therefore, the scope of this ADR is to address only point 1. above: Ensure broadcast messages are eventually received by all peers, given the sender does not stop before. ### Discussion * We are currently using the [ouroboros-framework](https://github.com/input-output-hk/ouroboros-network) and [typed-protocols](https://github.com/input-output-hk/typed-protocols) network stack as a mere [transport](https://osi-model.com/transport-layer/) layer. - * Being built on top of TCP, ourboros multiplexer (Mux) provides the same reliability guarantees, plus the multiplexing capabilities of course - * It also takes care of reconnecting to peers when a failure is detected which relieves us of doing so, but any reconnection implies a reset of each peer's state machine which means we need to make sure any change to the state of pending/received messages is handled by the applicative layer + * Being built on top of TCP, ouroboros multiplexer (Mux) provides the same reliability guarantees, plus the multiplexing capabilities of course + * It also takes care of reconnecting to peers when a failure is detected which relieves us from doing so, but any reconnection implies a reset of each peer's state machine which means we need to make sure any change to the state of pending/received messages is handled by the applicative layer * Our [FireForget](https://github.com/input-output-hk/hydra/blob/8a8e0829964132bde8949e5249a1ab303af92fb8/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L31) ignores connections/disconnections * Ouroboros/typed-protocols provides enough machinery to implement a reliable broadcast protocol, for example by reusing existing `KeepAlive` protocol and building a more robust point-to-point protocol than what we have now * There is a minor limitation, namely that the subscription mechanism does not handle connections invidually, but as a set of equivalent point-to-point full duplex connections whose size (valency) needs to be maintained at a certain threshold, which means that unless backed in the protocol itself, protocol state-machine and applications are not aware of the identity of the remote peer @@ -51,10 +51,14 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * That layer consumes and produces `Authenticate`d messages in order to identify the source of messages * It uses a vector of monotonically increasing _sequence numbers_ associated with each party (including itself) to track what are the last messages from each party and to ensure FIFO delivery of messages * This _vector_ is also used to identify peers which are lagging behind and to resend the missing messages, or to drop messages which have already been received +* This _vector_ is also piggybacked by the ping messages of the heartbeat mechanism so that a node informs its peers of its own situation even if it would have, otherwise, no message to sent to them +* Sending a ping message does not influence the _vector_ of the sender +* Any message received which index does not match what the peer expects is dropped +* Messages deemed not received by a peer a re-sent ## Consequences -* We keep our existing `Network` interface hence all message will be resent to all peers +* We keep our existing `Network` interface hence all messages will be resent to all peers * This could be later optimized either by providing a smarter interface with a `send :: Peer -> msg -> m ()` unicast function, or by adding a layer with filtering capabilities, or both * We need to ensure messages are not kept forever when resending, eg. that the pending messages list is garbage collected * We need to refactor our `Heartbeat` layer to decouple the 2 capabilities it embeds, namely sending periodical heartbeats to peers, and providing listeners with information about a peer connection status, so that the `Reliability` layer can actually rely on those heartbeats to get regular update of peers' knowledge even when the actual Head is idle. From 263d25f187d20564fadfb004de6df154eb2687be Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 10:17:27 +0200 Subject: [PATCH 04/14] FIX typo Co-authored-by: Sasha Bogicevic --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 387000c2d99..7447a2e3dc8 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -19,7 +19,7 @@ We want Hydra to be _Consistent_ in the presence of _Network Partitions_, under We have identified 3 main sources of failures in the _fail-recovery_ model that can lead to a head being stuck: 1. The network layer can drop messages from the moment a node `broadcast`s it, leading to some messages not being received at the other end -2. The sending node can crash in between the moment the state is changed (and persisted) and the moment a message is actually sent through the network (or even it calls `broadcast`) +2. The sending node can crash in between the moment the state is changed (and persisted) and the moment a message is actually sent through the network (or even when it calls `broadcast`) 3. The receiving node can crash in between the moment the message has been received in the network layer, and it's processed (goes through the queue) We agree that we'll want to address all those issues in order to provide a good user experience, as not addressing 2. and 3. can lead to hard to troubleshoot issues with heads. We have not experienced those issues yet as they would probably only crop up under heavy loads, or in the wild. But we also agree we want to tackle 1. first because it's where most of the risk lies. By providing a _Reliable Broadcast_ layer, we will significantly reduce the risks and can then later on address the other points. From a74fc67bd8fa1a4d1033ac975c35a6cf8758bcc4 Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 13:52:47 +0200 Subject: [PATCH 05/14] Fix status word --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 7447a2e3dc8..4f523f654fc 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -8,7 +8,7 @@ tags: [Proposed] ## Status -Proposed +Draft ## Context From 8050a22cea09d15c64e152056e66d716cf3d1dc8 Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 13:53:20 +0200 Subject: [PATCH 06/14] FIX typo Co-authored-by: Sebastian Nagel --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 4f523f654fc..772a6d951d9 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -31,7 +31,7 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * We are currently using the [ouroboros-framework](https://github.com/input-output-hk/ouroboros-network) and [typed-protocols](https://github.com/input-output-hk/typed-protocols) network stack as a mere [transport](https://osi-model.com/transport-layer/) layer. * Being built on top of TCP, ouroboros multiplexer (Mux) provides the same reliability guarantees, plus the multiplexing capabilities of course * It also takes care of reconnecting to peers when a failure is detected which relieves us from doing so, but any reconnection implies a reset of each peer's state machine which means we need to make sure any change to the state of pending/received messages is handled by the applicative layer - * Our [FireForget](https://github.com/input-output-hk/hydra/blob/8a8e0829964132bde8949e5249a1ab303af92fb8/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L31) ignores connections/disconnections + * Our [FireForget protocol](https://github.com/input-output-hk/hydra/blob/8a8e0829964132bde8949e5249a1ab303af92fb8/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L31) ignores connections/disconnections * Ouroboros/typed-protocols provides enough machinery to implement a reliable broadcast protocol, for example by reusing existing `KeepAlive` protocol and building a more robust point-to-point protocol than what we have now * There is a minor limitation, namely that the subscription mechanism does not handle connections invidually, but as a set of equivalent point-to-point full duplex connections whose size (valency) needs to be maintained at a certain threshold, which means that unless backed in the protocol itself, protocol state-machine and applications are not aware of the identity of the remote peer * We have built our `Network` infrastructure over the concept of relatively independent layers, each implementing a similar interface with different kind of messages, to `broadcast` messages to all peers and be notified of incoming messages through a `callback`. From ccb6f94b62325828cede2363433dd31544bc3357 Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 13:55:37 +0200 Subject: [PATCH 07/14] link to Ouroboros exemple --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 772a6d951d9..af37cb9df97 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -32,7 +32,7 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * Being built on top of TCP, ouroboros multiplexer (Mux) provides the same reliability guarantees, plus the multiplexing capabilities of course * It also takes care of reconnecting to peers when a failure is detected which relieves us from doing so, but any reconnection implies a reset of each peer's state machine which means we need to make sure any change to the state of pending/received messages is handled by the applicative layer * Our [FireForget protocol](https://github.com/input-output-hk/hydra/blob/8a8e0829964132bde8949e5249a1ab303af92fb8/hydra-node/src/Hydra/Network/Ouroboros/Type.hs#L31) ignores connections/disconnections - * Ouroboros/typed-protocols provides enough machinery to implement a reliable broadcast protocol, for example by reusing existing `KeepAlive` protocol and building a more robust point-to-point protocol than what we have now + * Ouroboros/typed-protocols provides enough machinery to implement a reliable broadcast protocol, for example by reusing existing `[KeepAlive](https://github.com/input-output-hk/ouroboros-network/tree/master/ouroboros-network-protocols/src/Ouroboros/Network/Protocol/KeepAlive)` protocol and building a more robust point-to-point protocol than what we have now * There is a minor limitation, namely that the subscription mechanism does not handle connections invidually, but as a set of equivalent point-to-point full duplex connections whose size (valency) needs to be maintained at a certain threshold, which means that unless backed in the protocol itself, protocol state-machine and applications are not aware of the identity of the remote peer * We have built our `Network` infrastructure over the concept of relatively independent layers, each implementing a similar interface with different kind of messages, to `broadcast` messages to all peers and be notified of incoming messages through a `callback`. * This pipes-like abstraction allows us to compose our network stack like: From f50f66bc18326f431f74371653a540baba197ebb Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 13:56:18 +0200 Subject: [PATCH 08/14] clarification Co-authored-by: Sebastian Nagel --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index af37cb9df97..97efee7ce03 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -43,7 +43,7 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad withOuroborosNetwork (contramap Network tracer) localhost peers ``` - * This has the nice property that we can basically swap the lower layers should we need to, for example to use [UDP](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/UDP.hs), or add other layers for example to identify [Multiple Heads](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/MultiHead.hs#L26) + * This has the nice property that we can basically swap the lower layers should we need to, for example to use [UDP](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/UDP.hs), or add other layers for example to address specific head instances in presence of [multiple heads](https://github.com/input-output-hk/hydra/blob/abailly-iohk/multi-node-udp/hydra-node/src/Hydra/Network/MultiHead.hs#L26) ## Decision From 671cf61a433675558100d2025445548126152992 Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 13:58:55 +0200 Subject: [PATCH 09/14] Clarification Co-authored-by: Sebastian Nagel --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 97efee7ce03..52e86b87b9b 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -48,7 +48,7 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad ## Decision * We implement our own message tracking and resending logic as a standalone `Network` layer -* That layer consumes and produces `Authenticate`d messages in order to identify the source of messages +* That layer consumes and produces `Authenticated msg` messages as it relies on identifying the source of messages * It uses a vector of monotonically increasing _sequence numbers_ associated with each party (including itself) to track what are the last messages from each party and to ensure FIFO delivery of messages * This _vector_ is also used to identify peers which are lagging behind and to resend the missing messages, or to drop messages which have already been received * This _vector_ is also piggybacked by the ping messages of the heartbeat mechanism so that a node informs its peers of its own situation even if it would have, otherwise, no message to sent to them From 4fd3b56e0db1e59ff9da9b39e2a41af6f7075ecf Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 13:59:12 +0200 Subject: [PATCH 10/14] FIX tyop Co-authored-by: Sebastian Nagel --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 52e86b87b9b..13f0e280e01 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -54,7 +54,7 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * This _vector_ is also piggybacked by the ping messages of the heartbeat mechanism so that a node informs its peers of its own situation even if it would have, otherwise, no message to sent to them * Sending a ping message does not influence the _vector_ of the sender * Any message received which index does not match what the peer expects is dropped -* Messages deemed not received by a peer a re-sent +* Messages deemed not received by a peer are re-sent ## Consequences From b6f7c7026302143d02622003e93ceb79945a7a42 Mon Sep 17 00:00:00 2001 From: Pascal Grange Date: Mon, 25 Sep 2023 14:03:54 +0200 Subject: [PATCH 11/14] These two items rather qualify as "decisions" than as consequences --- docs/adr/2023-09-08_026-network-resilience.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 13f0e280e01..e396983e26d 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -55,6 +55,8 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * Sending a ping message does not influence the _vector_ of the sender * Any message received which index does not match what the peer expects is dropped * Messages deemed not received by a peer are re-sent +* We do not implement a _pull-based_ message communication mechanism as initially envisioned +* We do not persist messages either on the receiving or sending side at this time ## Consequences @@ -63,6 +65,4 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * We need to ensure messages are not kept forever when resending, eg. that the pending messages list is garbage collected * We need to refactor our `Heartbeat` layer to decouple the 2 capabilities it embeds, namely sending periodical heartbeats to peers, and providing listeners with information about a peer connection status, so that the `Reliability` layer can actually rely on those heartbeats to get regular update of peers' knowledge even when the actual Head is idle. * We want to specify this protocol clearly in order to ease implementation in other languages, detailing the structure of messages and the semantics of retries and timeouts. -* We do not implement a _pull-based_ message communication mechanism as initially envisioned -* We do not persist messages either on the receiving or sending side at this time * We may consider relying on the vector clock in the future to ensure perfect ordering of messages on each peer and make impossible for legit transactions to be temporarily seen as invalid. This can happen in the current version and is handled through wait and ttl From fd781c8c61e191fb0d61f780b0dd4da8f4ad44d4 Mon Sep 17 00:00:00 2001 From: Arnaud Bailly Date: Mon, 25 Sep 2023 21:16:18 +0200 Subject: [PATCH 12/14] Remove some implementation details --- docs/adr/2023-09-08_026-network-resilience.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index e396983e26d..8faa89be2a2 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -50,11 +50,8 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * We implement our own message tracking and resending logic as a standalone `Network` layer * That layer consumes and produces `Authenticated msg` messages as it relies on identifying the source of messages * It uses a vector of monotonically increasing _sequence numbers_ associated with each party (including itself) to track what are the last messages from each party and to ensure FIFO delivery of messages -* This _vector_ is also used to identify peers which are lagging behind and to resend the missing messages, or to drop messages which have already been received -* This _vector_ is also piggybacked by the ping messages of the heartbeat mechanism so that a node informs its peers of its own situation even if it would have, otherwise, no message to sent to them -* Sending a ping message does not influence the _vector_ of the sender -* Any message received which index does not match what the peer expects is dropped -* Messages deemed not received by a peer are re-sent + * This _vector_ is used to identify peers which are lagging behind, resend the missing messages, or to drop messages which have already been received + * The _Heartbeat_ mechanism is relied upon to ensure dissemination of state even when the node is quiescent * We do not implement a _pull-based_ message communication mechanism as initially envisioned * We do not persist messages either on the receiving or sending side at this time From 6d5a0fdcee8c4024760e2d0e3434dfc094e1a7ad Mon Sep 17 00:00:00 2001 From: Arnaud Bailly Date: Mon, 25 Sep 2023 21:17:51 +0200 Subject: [PATCH 13/14] Remove note about garbage collection This seems an implementation detail --- docs/adr/2023-09-08_026-network-resilience.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 8faa89be2a2..2db604141a3 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -59,7 +59,5 @@ Therefore, the scope of this ADR is to address only point 1. above: Ensure broad * We keep our existing `Network` interface hence all messages will be resent to all peers * This could be later optimized either by providing a smarter interface with a `send :: Peer -> msg -> m ()` unicast function, or by adding a layer with filtering capabilities, or both -* We need to ensure messages are not kept forever when resending, eg. that the pending messages list is garbage collected -* We need to refactor our `Heartbeat` layer to decouple the 2 capabilities it embeds, namely sending periodical heartbeats to peers, and providing listeners with information about a peer connection status, so that the `Reliability` layer can actually rely on those heartbeats to get regular update of peers' knowledge even when the actual Head is idle. * We want to specify this protocol clearly in order to ease implementation in other languages, detailing the structure of messages and the semantics of retries and timeouts. -* We may consider relying on the vector clock in the future to ensure perfect ordering of messages on each peer and make impossible for legit transactions to be temporarily seen as invalid. This can happen in the current version and is handled through wait and ttl +* We may consider relying on the vector clock in the future to ensure perfect ordering of messages on each peer and make impossible for legit transactions to be temporarily seen as invalid. This can happen in the current version and is handled through wait and _TTL_ From 17c0a24c3c407f9f5faa32677f61bcd5720fd543 Mon Sep 17 00:00:00 2001 From: Arnaud Bailly Date: Mon, 25 Sep 2023 21:19:16 +0200 Subject: [PATCH 14/14] Added authors --- docs/adr/2023-09-08_026-network-resilience.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/adr/2023-09-08_026-network-resilience.md b/docs/adr/2023-09-08_026-network-resilience.md index 2db604141a3..e3d0999b2f6 100644 --- a/docs/adr/2023-09-08_026-network-resilience.md +++ b/docs/adr/2023-09-08_026-network-resilience.md @@ -2,7 +2,7 @@ slug: 26 title: | 26. Network failures model -authors: [] +authors: [abailly, pgrange] tags: [Proposed] ---