diff --git a/content/concepts/publish-subscribe.md b/content/concepts/publish-subscribe.md new file mode 100644 index 00000000..7ca1a8ee --- /dev/null +++ b/content/concepts/publish-subscribe.md @@ -0,0 +1,309 @@ +--- +title: Publish/Subscribe +--- + +Publish/Subscribe is a system where peers congregate around topics they are +interested in. Peers interested in a topic are said to be subscribed to that +topic: + +![](subscribed_peers.png) + +Peers can send messages to topics. Each message gets delivered to all peers +subscribed to the topic: + +![](message_delivered_to_all.png) + +Example uses of pub/sub: + +* **Chat rooms.** Each room is a pub/sub topic and clients post chat messages to + which are received by all other clients in the room. +* **File sharing.** Each pub/sub topic represents a file that can be downloaded. + Uploaders and downloaders advertise which pieces of the file they have in + the pub/sub topic and coordinate downloads that will happen outside the + pub/sub system. + +## Design goals + +In a peer-to-peer pub/sub system all peers participate in delivering messages +throughout the network. There are several different designs for peer-to-peer +pub/sub systems which offer different trade-offs. Desirable properties include: + +* **Reliability:** All messages get delivered to all peers subscribed to the topic. +* **Speed:** Messages are delivered quickly. +* **Efficiency:** The network is not flooded with excess copies of messages. +* **Resilience:** Peers can join and leave the network without disrupting it. + There is no central point of failure. +* **Scale:** Topics can have enormous numbers of subscribers and handle a large + throughput of messages. +* **Simplicity:** The system is simple to understand and implement. Each peer + only needs to remember a small amount of state. + +libp2p currently uses a design called **gossipsub**. It is named after the fact +that peers gossip to each other about which messages they have seen and use this +information to maintain a message delivery network. + +## Discovery + +Before a peer can subscribe to a topic is must find other peers and establish +network connections with them. The pub/sub system doesn’t have any way to +discover peers by itself. Instead, it relies upon the application to find new +peers on its behalf, a process called **ambient peer discovery**. + +Potential methods for discovering peers include: + +* Distributed hash tables +* Local network broadcasts +* Exchanging peer lists with existing peers +* Centralized trackers or rendezvous points +* Lists of bootstrap peers + +For example, in a BitTorrent application, most of the above methods would +already be used in the process of downloading files. By reusing peers found +while the BitTorrent application goes about its regular business, the +application could build up a robust pub/sub network too. + +Discovered peers are asked if they support the pub/sub protocol, and if so, are +added to the pub/sub network. + +## Types of peering + +In gossipsub, peers connect to each other via either **full-message** peerings +or **metadata-only** peerings. The overall network structure is made up of these +two networks: + +![](types_of_peering.png) + +### Full-message + +Full-message peerings are used to transmit the full contents of messages +throughout the network. This network is sparsely-connected with each peer only +being connected to a few other peers. (In the [gossipsub specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md) +this sparsely-connected network is called a *mesh* and peers within it are +called *mesh members*.) + +Limiting the number of full-message peerings is useful because it keeps the +amount of network traffic under control; each peer only forwards messages to a +few other peers, rather than all of them. Each peer has a target number of peers +it wants to be connected to. In this example each peer would ideally like to be +connected to 3 other peers, but would settle +for 2–4 +connections: + +![](full_message_network.png) + +

Throughout this guide, numbers +highlighted in purple can be configured +by the developer.

+ +The peering degree (also called the *network degree* or *D*) controls the +trade-off between speed, reliability, resilience and efficiency of the network. +A higher peering degree helps messages get delivered faster, with a better +chance of reaching all subscribers and with less chance of any peer disrupting +the network by leaving. However, a high peering degree also causes additional +redundant copies of each message to be sent throughout the network, increasing +the bandwidth required to participate in the network. + +In libp2p’s default implementation the ideal network peering degree is +6 with anywhere from +4–12 +being acceptable. + +### Metadata-only + +In addition to the sparsely-connected network of full-message peerings, there is +also a densely-connected network of metadata-only peerings. This network is made +up of all the network connections between peers that aren’t full-message +peerings. + +The metadata-only network shares gossip about which messages are available and +performs functions to help maintain the network of full-message peerings. + +![](metadata_only_network.png) + +## Grafting and pruning + +Peerings are **bidirectional**, meaning that for any two connected peers, both +peers consider their connection to be full-message or both peers consider their +connection to be metadata-only. + +Either peer can change the connection type by notifying the other. **Grafting** is +the process of converting a metadata-only connection to full-message. **Pruning** +is the opposite process; converting a full-message peering to metadata-only: + +![](graft_prune.png) + +When a peer has too few full-message peerings it will randomly graft some of its +metadata-only peerings to become full-message peerings: + +![](maintain_graft.png) + +Conversely, when a peer has too many full-message peerings it will randomly +prune some of them back to metadata-only: + +![](maintain_prune.png) + +In libp2p’s implementation, each peer performs a series of checks every +1 second. These checks are called the +*heartbeat*. Grafting and pruning happens during this time. + +## Subscribing and unsubscribing + +Peers keep track of which topics their directly-connected peers are subscribed +to. Using this information each peer is able to build up a picture of the topics +around them and which peers are subscribed to each topic: + +![](subscriptions_local_view.png) + +Keeping track of subscriptions happens by sending **subscribe** and +**unsubscribe** messages. When a new connection is established between two peers +they start by sending each other the list of topics they are subscribed to: + +![](subscription_list_first_connect.png) + +Then over time, whenever a peer subscribes or unsubscribes from a topic, it will +send each of its peers a subscribe or unsubscribe message. These messages are +sent to all connected peers regardless of whether the receiving peer is +subscribed to the topic in question: + +![](subscription_list_change.png) + +Subscribe and unsubscribe messages go hand-in-hand with graft and prune +messages. When a peer subscribes to a topic it will pick some peers that will +become its full-message peers for that topic and send them graft messages at the +same time as their subscribe messages: + +![](subscribe_graft.png) + +When a peer unsubscribes from a topic it will notify its full-message peers that +their connection has been pruned at the same time as sending their unsubscribe +messages: + +![](unsubscribe_prune.png) + +## Sending messages + +When a peer wants to publish a message it sends a copy to all full-message peers +it is connected to: + +![](full_message_send.png) + +Similarly, when a peer receives a new message from another peer, it stores the +message and forwards a copy to all other full-message peers it is connected to: + +![](full_message_forward.png) + +In the [gossipsub specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md#controlling-the-flood), +peers are also known as *routers* because of this function they have in routing +messages through the network. + +Peers remember a list of recently seen messages. This lets peers act upon a +message only the first time they see it and ignore retransmissions of already +seen messages. + +Peers might also choose to validate the contents of each message received. What +counts as valid and invalid depends on the application. For example, a chat +application might enforce that all messages must be shorter than 100 characters. +If the application tells libp2p that a message is invalid then that message will +be dropped and not replicated further through the network. + +## Gossip + +Peers gossip about messages they have recently seen. Every +1 second each peer randomly selects +6 metadata-only peers and sends them a list of +recently seen messages. + +![](gossip_deliver.png) + +Gossiping gives peers a chance to notice in case they missed a message on the +full-message network. If a peer notices it is repeatedly missing messages then +it can set up new full-message peerings with peers that do have the messages. + +Here is an example of how a specific message can be requested across a +metadata-only peering: + +![](request_gossiped_message.png) + +In the [gossipsub specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md#control-messages), +gossip announcing recently seen messages are called *IHAVE* messages and +requests for specific messages are called *IWANT* messages. + +## Fan-out + +Peers are allowed to publish messages to topics they are not subscribed to. +There are some special rules about how to do this to help ensure these messages +are delivered reliably. + +The first time a peer wants to publish a message to a topic it is not subscribed +to, it randomly picks 6 peers +(3 shown below) that are +subscribed to that topic and remembers them as **fan-out** peers for that topic: + +![](fanout_initial_pick.png) + +Unlike the other types of peering, fan-out peerings are unidirectional; they +always point from the peer outside the topic to a peer subscribed to the topic. +Peers subscribed to the topic are not told that they have been selected +and still treat the connection as any other metadata-only peering. + +Each time the sender wants to send a message, it sends the message to its +fan-out peers, who then distribute the message within the topic: + +![](fanout_message_send.png) + +If the sender goes to send a message but notices some of their fan-out peers +went away since last time, they will randomly select additional fan-out peers +to top them back up to 6. + +When a peer subscribes to a topic, if it already has some fan-out peers it will +prefer them to become the full-message peers: + +![](fanout_grafting_preference.png) + +After 2 minutes of not sending any messages to +a topic, all the fan-out peers for that topic are forgotten: + +![](fanout_forget.png) + +## Network packets + +The packets that peers actually send each other over the network are a +combination of all the different message types seen in this guide (application +messages, have/want, subscribe/unsubscribe, graft/prune). This structure allows +several different requests to be batched up and sent in a single network packet. + +Here is a graphical representation of the overall network packet structure: + +![](network_packet_structure.png) + +See the [specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md#protobuf) for the exact [Protocol Buffers](https://developers.google.com/protocol-buffers) +schema used to encode network packets. + +## State + +Here is a summary of the state each peer must remember to participate in the +pub/sub network: + +* **Subscriptions:** List of topics subscribed to. +* **Fan-out topics:** These are the topics messaged recently but not subscribed + to. For each topic the time of the last sent message to that topic is + remembered. +* **List of peers currently connected to:** For each peer connected to, the + state includes all the topics they are subscribed to and whether the peering + for each topic is full-content, metadata-only or fan-out. +* **Recently seen messages**: This is a cache of recently seen messages. It is + used to detect and ignore retransmitted messages. For each message the state + includes who sent it and the sequence number, which is enough to uniquely + identify any message. For very recent messages, the full message contents + is kept so that it can be sent to any peers that request the message. + +![](state.png) + +## More information + +For more detail and a discussion of other pub/sub designs that influenced the +design of gossipsub, see the [gossipsub specification](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/README.md). + +For implementation details, see the [gossipsub.go](https://github.com/libp2p/go-libp2p-pubsub/blob/master/gossipsub.go) +file in the source code of [go-libp2p-pubsub](https://github.com/libp2p/go-libp2p-pubsub), +which is the canonical implementation of gossipsub in libp2p. diff --git a/content/concepts/publish-subscribe/fanout_forget.png b/content/concepts/publish-subscribe/fanout_forget.png new file mode 100644 index 00000000..0bec2d3c Binary files /dev/null and b/content/concepts/publish-subscribe/fanout_forget.png differ diff --git a/content/concepts/publish-subscribe/fanout_forget.svg b/content/concepts/publish-subscribe/fanout_forget.svg new file mode 100644 index 00000000..d0dcc13c --- /dev/null +++ b/content/concepts/publish-subscribe/fanout_forget.svg @@ -0,0 +1,946 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + After not publishing anymessages for 2 minutes… + + All fan-out peerings revert tometadata-only peerings + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/concepts/publish-subscribe/fanout_grafting_preference.png b/content/concepts/publish-subscribe/fanout_grafting_preference.png new file mode 100644 index 00000000..babd8e91 Binary files /dev/null and b/content/concepts/publish-subscribe/fanout_grafting_preference.png differ diff --git a/content/concepts/publish-subscribe/fanout_grafting_preference.svg b/content/concepts/publish-subscribe/fanout_grafting_preference.svg new file mode 100644 index 00000000..02d27db7 --- /dev/null +++ b/content/concepts/publish-subscribe/fanout_grafting_preference.svg @@ -0,0 +1,876 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + Peer has existingfan-out peerings + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Upon subscribing to the topic,fan-out peerings preferentiallybecome full-message peerings + + + + + + + diff --git a/content/concepts/publish-subscribe/fanout_initial_pick.png b/content/concepts/publish-subscribe/fanout_initial_pick.png new file mode 100644 index 00000000..b112dc42 Binary files /dev/null and b/content/concepts/publish-subscribe/fanout_initial_pick.png differ diff --git a/content/concepts/publish-subscribe/fanout_initial_pick.svg b/content/concepts/publish-subscribe/fanout_initial_pick.svg new file mode 100644 index 00000000..5404e356 --- /dev/null +++ b/content/concepts/publish-subscribe/fanout_initial_pick.svg @@ -0,0 +1,1015 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Peer wants to publish a messageto a topic it is not subscribed to + Randomly select 3 peers subscribed to thetopic and remember them as fan-out peers + + Full-message peering + + Fan-out peering + + Metadata-only peering + + + + diff --git a/content/concepts/publish-subscribe/fanout_message_send.png b/content/concepts/publish-subscribe/fanout_message_send.png new file mode 100644 index 00000000..45a15e05 Binary files /dev/null and b/content/concepts/publish-subscribe/fanout_message_send.png differ diff --git a/content/concepts/publish-subscribe/fanout_message_send.svg b/content/concepts/publish-subscribe/fanout_message_send.svg new file mode 100644 index 00000000..d3a28b1b --- /dev/null +++ b/content/concepts/publish-subscribe/fanout_message_send.svg @@ -0,0 +1,999 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + New message sentto fan-out peers + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Once inside the topic, the message isforwarded to all other subscribers as usual + + diff --git a/content/concepts/publish-subscribe/full_message_forward.png b/content/concepts/publish-subscribe/full_message_forward.png new file mode 100644 index 00000000..90ebab00 Binary files /dev/null and b/content/concepts/publish-subscribe/full_message_forward.png differ diff --git a/content/concepts/publish-subscribe/full_message_forward.svg b/content/concepts/publish-subscribe/full_message_forward.svg new file mode 100644 index 00000000..5d609174 --- /dev/null +++ b/content/concepts/publish-subscribe/full_message_forward.svg @@ -0,0 +1,852 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Incomingmessage + + + + + + + Peer stores a copyof the message + + + + + + + + + + + + + + + + + + + + + + + + + + + + Message forwarded to all other full-message peers + + diff --git a/content/concepts/publish-subscribe/full_message_network.png b/content/concepts/publish-subscribe/full_message_network.png new file mode 100644 index 00000000..f3205799 Binary files /dev/null and b/content/concepts/publish-subscribe/full_message_network.png differ diff --git a/content/concepts/publish-subscribe/full_message_network.svg b/content/concepts/publish-subscribe/full_message_network.svg new file mode 100644 index 00000000..bf9846a0 --- /dev/null +++ b/content/concepts/publish-subscribe/full_message_network.svg @@ -0,0 +1,1591 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Network peering degree = 3 + + + + + + + Peer reachedupper bound + Peer reachedlower bound + Lower bound = 2 + + + + + + + + + + + + + Upper bound = 4 + Full-message network + + diff --git a/content/concepts/publish-subscribe/full_message_send.png b/content/concepts/publish-subscribe/full_message_send.png new file mode 100644 index 00000000..934748e9 Binary files /dev/null and b/content/concepts/publish-subscribe/full_message_send.png differ diff --git a/content/concepts/publish-subscribe/full_message_send.svg b/content/concepts/publish-subscribe/full_message_send.svg new file mode 100644 index 00000000..fa52520d --- /dev/null +++ b/content/concepts/publish-subscribe/full_message_send.svg @@ -0,0 +1,800 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Peer creates newmessage of its own + + + + + + + + + + + + + + + + + + + + + Message sent to all other full-message peers + + + + + + diff --git a/content/concepts/publish-subscribe/gossip_deliver.png b/content/concepts/publish-subscribe/gossip_deliver.png new file mode 100644 index 00000000..9fa1a2c5 Binary files /dev/null and b/content/concepts/publish-subscribe/gossip_deliver.png differ diff --git a/content/concepts/publish-subscribe/gossip_deliver.svg b/content/concepts/publish-subscribe/gossip_deliver.svg new file mode 100644 index 00000000..860e63e1 --- /dev/null +++ b/content/concepts/publish-subscribe/gossip_deliver.svg @@ -0,0 +1,1476 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + I have seen: + + + + + + + I have seen: + + + + + + + I have seen: + + + + + + + I have seen: + + + + + + + I have seen: + + + + + + + I have seen: + + + + + + + + Every 1 second… + + + Select 6 metadata-onlypeerings at random + + + + + + + + + + + + + + + + + + + + + Send them a list ofrecently seen messages + + + + + + + + + + + + + + Seen messages specify thesender and sequence number,but not the full message contents + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/concepts/publish-subscribe/graft_prune.png b/content/concepts/publish-subscribe/graft_prune.png new file mode 100644 index 00000000..23fa86c4 Binary files /dev/null and b/content/concepts/publish-subscribe/graft_prune.png differ diff --git a/content/concepts/publish-subscribe/graft_prune.svg b/content/concepts/publish-subscribe/graft_prune.svg new file mode 100644 index 00000000..91b9924a --- /dev/null +++ b/content/concepts/publish-subscribe/graft_prune.svg @@ -0,0 +1,280 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + I’m grafting ourconnection into afull-message peering. + + + + + + + + + + Full-message peering + + + + + I’m pruning ourconnection back to ametadata-only peering. + + + + + + + + + + Metadata-only peering + + + diff --git a/content/concepts/publish-subscribe/maintain_graft.png b/content/concepts/publish-subscribe/maintain_graft.png new file mode 100644 index 00000000..c869a867 Binary files /dev/null and b/content/concepts/publish-subscribe/maintain_graft.png differ diff --git a/content/concepts/publish-subscribe/maintain_graft.svg b/content/concepts/publish-subscribe/maintain_graft.svg new file mode 100644 index 00000000..47f717ab --- /dev/null +++ b/content/concepts/publish-subscribe/maintain_graft.svg @@ -0,0 +1,1168 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Too few + Too many + Acceptable amount + Ideal + + Start with 2full-contentpeerings + + + + + + + + + + + + + + + + + Ideal + + Select more peers to graftto get to the ideal number + + + + + + + + + + + + + + + + + + Ideal + + Grafting complete + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Too few + Too many + Acceptable amount + + + + Too few + Too many + Acceptable amount + + diff --git a/content/concepts/publish-subscribe/maintain_prune.png b/content/concepts/publish-subscribe/maintain_prune.png new file mode 100644 index 00000000..2ff8bb6e Binary files /dev/null and b/content/concepts/publish-subscribe/maintain_prune.png differ diff --git a/content/concepts/publish-subscribe/maintain_prune.svg b/content/concepts/publish-subscribe/maintain_prune.svg new file mode 100644 index 00000000..e33c52c0 --- /dev/null +++ b/content/concepts/publish-subscribe/maintain_prune.svg @@ -0,0 +1,1325 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Too few + Too many + Acceptable amount + Ideal + + Start with 14full-contentpeerings + + + + + + + + + + + + + + + + + Ideal + + Select peers to prune toget to the ideal number + + + + + + + + + + + + + + + + + + Ideal + + Pruning complete + + + + Too few + Too many + Acceptable amount + + + + Too few + Too many + Acceptable amount + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/content/concepts/publish-subscribe/message_delivered_to_all.png b/content/concepts/publish-subscribe/message_delivered_to_all.png new file mode 100644 index 00000000..9898c7a8 Binary files /dev/null and b/content/concepts/publish-subscribe/message_delivered_to_all.png differ diff --git a/content/concepts/publish-subscribe/message_delivered_to_all.svg b/content/concepts/publish-subscribe/message_delivered_to_all.svg new file mode 100644 index 00000000..0911d972 --- /dev/null +++ b/content/concepts/publish-subscribe/message_delivered_to_all.svg @@ -0,0 +1,863 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Message + + + diff --git a/content/concepts/publish-subscribe/metadata_only_network.png b/content/concepts/publish-subscribe/metadata_only_network.png new file mode 100644 index 00000000..15601542 Binary files /dev/null and b/content/concepts/publish-subscribe/metadata_only_network.png differ diff --git a/content/concepts/publish-subscribe/metadata_only_network.svg b/content/concepts/publish-subscribe/metadata_only_network.svg new file mode 100644 index 00000000..85e6e1df --- /dev/null +++ b/content/concepts/publish-subscribe/metadata_only_network.svg @@ -0,0 +1,2316 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Metadata-only peering + + Each peering is a networkconnection between two peers + + + + Metadata-only network + + + diff --git a/content/concepts/publish-subscribe/network_packet_structure.png b/content/concepts/publish-subscribe/network_packet_structure.png new file mode 100644 index 00000000..eb6e6e1f Binary files /dev/null and b/content/concepts/publish-subscribe/network_packet_structure.png differ diff --git a/content/concepts/publish-subscribe/network_packet_structure.svg b/content/concepts/publish-subscribe/network_packet_structure.svg new file mode 100644 index 00000000..0d6ff79c --- /dev/null +++ b/content/concepts/publish-subscribe/network_packet_structure.svg @@ -0,0 +1,2358 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Recipient (Topic ID) + + Topic 1 + Sender + + Seq no. + 4 + + Peer D + Sender + + Seq no. + 3 + + Peer D + Sender + + Peer A + Recipients (Topic ID’s) + + Topic 1 + + Topic 3 + Message body + + The heaviest domestic cat on record is 21.297 kilograms. + Sender public key + Sender signature + Seq no. + + 5 + + + Application messages + + + + + I have seen these messages + + (IHAVE) + (IWANT) + + + + Please send me these messages + + + For this topic… + + + Subscription changes + + + For this topic… + + + Grafting and pruning + + + + Topic 1 + You have been… + + Pruned + + Grafted + + + + + Topic 6 + + Topic 7 + + Topic 4 + + Topic 6 + + Topic 7 + + + 133 + + 179 + + 70 + + 43 + + 69 + + 11 + + 130 + + 124 + + 170 + + … + + + + 111 + + 138 + + 217 + + 17 + + 106 + + 181 + + 66 + + 98 + + 59 + + … + + + + Topic 2 + + Topic 4 + + 10 + + Peer E + + Topic 5 + + 6 + + Peer F + + 14 + + Peer C + + + 1 + + Peer B + + Network packet + Application message + + Application message + + + + + + + + + + + + + + + + + + + + + Pruned + + Grafted + + + + + Pruned + + Grafted + + + + I want to… + + Unsub + + Subscribe + + + + + Unsub + + Subscribe + + + + + Unsub + + Subscribe + + + + + diff --git a/content/concepts/publish-subscribe/request_gossiped_message.png b/content/concepts/publish-subscribe/request_gossiped_message.png new file mode 100644 index 00000000..26fcd66b Binary files /dev/null and b/content/concepts/publish-subscribe/request_gossiped_message.png differ diff --git a/content/concepts/publish-subscribe/request_gossiped_message.svg b/content/concepts/publish-subscribe/request_gossiped_message.svg new file mode 100644 index 00000000..8ea60dc8 --- /dev/null +++ b/content/concepts/publish-subscribe/request_gossiped_message.svg @@ -0,0 +1,1725 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + I have seen: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Please send: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Peer receives amessage from theirfull-message peers + Peer waits until heartbeatand selects randommetadata-only peers + Newly received messageis gossiped to metadata-onlypeers + Peer notices that it doesnot have the gossipedmessage and requests it + Requested messageis transferred + Newly receivedmessage is broadcastto full-content peers + + diff --git a/content/concepts/publish-subscribe/state.png b/content/concepts/publish-subscribe/state.png new file mode 100644 index 00000000..6cdb6b6a Binary files /dev/null and b/content/concepts/publish-subscribe/state.png differ diff --git a/content/concepts/publish-subscribe/state.svg b/content/concepts/publish-subscribe/state.svg new file mode 100644 index 00000000..3580af03 --- /dev/null +++ b/content/concepts/publish-subscribe/state.svg @@ -0,0 +1,2137 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + Peer state + + + Topics I subscribe to + + + + + Topics I recently sent a message to + + + + + Peers currently connected to + + + + + Recently seen messages + + + Topic ID + + Topic 1 + + Topic 4 + + Topic 5 + + + + Topic ID + + Topic 6 + + Topic 7 + + + + + Time of last sent message + (but don’t subscribed to) + 10 seconds ago + 35 seconds ago + Peer ID + + Peer D + Topics they subscribe to + + Type of peering + Topic 1 + + Topic 4 + + + + Peer E + + Topic 5 + + Topic 6 + + + + Peer F + + Topic 7 + + + + Peer B + + + Metadata + Full + + + Topic 1 + + + Peer C + + + Topic 1 + + Sender (Peer ID) + + Peer D + + Seq no. + 2 + Time first seen + + 1 second ago + Full message + + + + + Last 2 minutes + + Last few seconds + + Peer E + + 5 + + 2 seconds ago + + + Peer F + + 10 + + 5 seconds ago + + + Peer E + + 4 + + 10 seconds ago + + + Peer D + + 1 + + 26 seconds ago + + Peer E + + 3 + + 31 seconds ago + + Peer E + + 2 + + 45 seconds ago + + Peer C + + 4 + + 90 seconds ago + + + + + + + + + + + + + + + + Fan-out + + + Metadata + Full + + + + + + Fan-out + + + Metadata + Full + + + + + + Fan-out + + + Metadata + Full + + + + + + Fan-out + + + Metadata + Full + + + + + + Fan-out + + + Metadata + Full + + + + + + Fan-out + + + Metadata + Full + + + + + + Fan-out + + + diff --git a/content/concepts/publish-subscribe/subscribe_graft.png b/content/concepts/publish-subscribe/subscribe_graft.png new file mode 100644 index 00000000..82851e71 Binary files /dev/null and b/content/concepts/publish-subscribe/subscribe_graft.png differ diff --git a/content/concepts/publish-subscribe/subscribe_graft.svg b/content/concepts/publish-subscribe/subscribe_graft.svg new file mode 100644 index 00000000..9294cb14 --- /dev/null +++ b/content/concepts/publish-subscribe/subscribe_graft.svg @@ -0,0 +1,313 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + I am subscribed to: + + + + + Topic 3 + + Also, I’m grafting ourconnection into afull-message peering. + + Topic 3 + + I am subscribed to: + + + + + Topic 3 + + + + + + + + Topic 3 + + Full-message peering + + diff --git a/content/concepts/publish-subscribe/subscribed_peers.png b/content/concepts/publish-subscribe/subscribed_peers.png new file mode 100644 index 00000000..cc2a9bac Binary files /dev/null and b/content/concepts/publish-subscribe/subscribed_peers.png differ diff --git a/content/concepts/publish-subscribe/subscribed_peers.svg b/content/concepts/publish-subscribe/subscribed_peers.svg new file mode 100644 index 00000000..82006243 --- /dev/null +++ b/content/concepts/publish-subscribe/subscribed_peers.svg @@ -0,0 +1,748 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Peer + + Topic + + + + + + + + + + + + + + Peers subscribedto topic + + + + + diff --git a/content/concepts/publish-subscribe/subscription_list_change.png b/content/concepts/publish-subscribe/subscription_list_change.png new file mode 100644 index 00000000..676c6d99 Binary files /dev/null and b/content/concepts/publish-subscribe/subscription_list_change.png differ diff --git a/content/concepts/publish-subscribe/subscription_list_change.svg b/content/concepts/publish-subscribe/subscription_list_change.svg new file mode 100644 index 00000000..363f5fcb --- /dev/null +++ b/content/concepts/publish-subscribe/subscription_list_change.svg @@ -0,0 +1,185 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + I am subscribed to: + + + + + Topic 5 + Topic 2 + Topic 3 + + + + I am unsubscribed from: + + diff --git a/content/concepts/publish-subscribe/subscription_list_first_connect.png b/content/concepts/publish-subscribe/subscription_list_first_connect.png new file mode 100644 index 00000000..1c19242c Binary files /dev/null and b/content/concepts/publish-subscribe/subscription_list_first_connect.png differ diff --git a/content/concepts/publish-subscribe/subscription_list_first_connect.svg b/content/concepts/publish-subscribe/subscription_list_first_connect.svg new file mode 100644 index 00000000..96e5dda5 --- /dev/null +++ b/content/concepts/publish-subscribe/subscription_list_first_connect.svg @@ -0,0 +1,247 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + I am subscribed to: + + + + + + + + + Topic 1 + Topic 2 + Topic 3 + + + + + I am subscribed to: + Topic 1 + Topic 5 + + + + diff --git a/content/concepts/publish-subscribe/subscriptions_local_view.png b/content/concepts/publish-subscribe/subscriptions_local_view.png new file mode 100644 index 00000000..8df865db Binary files /dev/null and b/content/concepts/publish-subscribe/subscriptions_local_view.png differ diff --git a/content/concepts/publish-subscribe/subscriptions_local_view.svg b/content/concepts/publish-subscribe/subscriptions_local_view.svg new file mode 100644 index 00000000..dc3f7573 --- /dev/null +++ b/content/concepts/publish-subscribe/subscriptions_local_view.svg @@ -0,0 +1,750 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Topic 1 + Topic 5 + Topic 3 + Topic 4 + Topic 2 + + + Viewing from this peer’s perspective + Directly-connected peer + + Topic + + + + + + + + + + Peers keep track ofothers’ subscriptionseven if not subscribedto the same topicsas them + + + + diff --git a/content/concepts/publish-subscribe/types_of_peering.png b/content/concepts/publish-subscribe/types_of_peering.png new file mode 100644 index 00000000..f977d04f Binary files /dev/null and b/content/concepts/publish-subscribe/types_of_peering.png differ diff --git a/content/concepts/publish-subscribe/types_of_peering.svg b/content/concepts/publish-subscribe/types_of_peering.svg new file mode 100644 index 00000000..29b70976 --- /dev/null +++ b/content/concepts/publish-subscribe/types_of_peering.svg @@ -0,0 +1,2319 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Full-message peering + + Metadata-only peering + + + Peer + + Topic + + diff --git a/content/concepts/publish-subscribe/unsubscribe_prune.png b/content/concepts/publish-subscribe/unsubscribe_prune.png new file mode 100644 index 00000000..cdd147df Binary files /dev/null and b/content/concepts/publish-subscribe/unsubscribe_prune.png differ diff --git a/content/concepts/publish-subscribe/unsubscribe_prune.svg b/content/concepts/publish-subscribe/unsubscribe_prune.svg new file mode 100644 index 00000000..24b889bc --- /dev/null +++ b/content/concepts/publish-subscribe/unsubscribe_prune.svg @@ -0,0 +1,324 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + Topic 3 + + + + + + + Topic 3 + + Full-message peering + + + I am unsubscribed from: + + + + + Topic 3 + + Also, I’m pruning ourconnection back to ametadata-only peering. + + I am unsubscribed from: + + + + + Topic 3 + + Metadata-only peering + + diff --git a/static/css/theme-libp2p.css b/static/css/theme-libp2p.css index 56f701c7..890ab8dc 100644 --- a/static/css/theme-libp2p.css +++ b/static/css/theme-libp2p.css @@ -7,8 +7,8 @@ --MAIN-LINK-HOVER-color:#167ad0; /* Color of hovered links */ --MAIN-ANCHOR-color: #1C90F3; /* color of anchors on titles */ - --MENU-HEADER-BG-color:#f3f6f9; /* Background color of menu header */ - --MENU-HEADER-BORDER-color:#EF65A4; /*Color of menu header border */ + --MENU-HEADER-BG-color:#EBF0F5; /* Background color of menu header */ + --MENU-HEADER-BORDER-color:#F065A4; /*Color of menu header border */ --MENU-SEARCH-BG-color:#fff; /* Search field background color (by default borders + icons) */ --MENU-SEARCH-BOX-color: #000; /* Override search field border color */ diff --git a/static/css/theme.css b/static/css/theme.css index 407fbeb8..7fbcce0d 100644 --- a/static/css/theme.css +++ b/static/css/theme.css @@ -623,7 +623,7 @@ code, kbd, pre, samp { font-family: "Consolas", menlo, monospace; font-size: 92%; } -code { +code, .configurable { border-radius: 2px; white-space: nowrap; color: #5e5e5e; @@ -631,6 +631,10 @@ code { border: 1px solid #fbf0cb; padding: 0px 2px; } +.configurable { + background: #eed9ee; + border-color: #e1c5e2; +} code + .copy-to-clipboard { margin-left: -1px; border-left: 0 !important; diff --git a/static/images/style_guide_1.svg b/static/images/style_guide_1.svg new file mode 100644 index 00000000..83690a58 --- /dev/null +++ b/static/images/style_guide_1.svg @@ -0,0 +1,1774 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Overview TutorialsExamples Reference + Specifications &Planning + At the core of libp2p is a set of specifications, which togetherform the definition for what libp2p is in the abstract and whatmakes a “correct” libp2p implementation. + Today, implementations of libp2p exist in several languages,with varying degrees of completeness. + Work Sans Medium40 px / 1.1#14152d + Work Sans Medium32 px / 1.1#14152d + IBM Plex Sans Normal16 px / 1.75#37384 + + 45 px + + 28 px + + + + + + + + Main palette + + + + Shading + Inverted background + Hyperlink + Light shading + + + + + Reserve palette + + 7.7.7.7:6542 + IBM Plex Mono Normal15 px / 1.15#5e5e5e + /p2p/QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWj/ip4/7.7.7.7/tcp/4242/p2p/QmYyQSo1c1Ym7orWxLYvC + + 7.7.7.7:6542 + + 7.7.7.7:6542 + + diff --git a/static/images/style_guide_2.svg b/static/images/style_guide_2.svg new file mode 100644 index 00000000..c8624f6a --- /dev/null +++ b/static/images/style_guide_2.svg @@ -0,0 +1,518 @@ + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + Diagram elements + Label textLabel textLabel text + IBM Plex Sans12 px / 1.25#37384 + Label + + 0.6 px line thickness5 px offset from text anchor + + + Arrows + + + Peer with arrow spacing + + Topic + + Message + + + + + + + Progression arrow + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +