Merge pull request #45 from PowerLoom/new_proto

Protocol V2 docs

docs/Protocol/Protocol_v2/_category_.json

@@ -0,0 +1,8 @@
+{
+    "label": "Protocol v2",
+    "position":1,
+    "link": {
+      "type": "generated-index",
+      "description": "Dive into the technical details of the new components and features of the Protocol that makes it more scalable, lightweight and decentralized."
+    }
+  }

docs/Protocol/Protocol_v2/overview.md

@@ -0,0 +1,70 @@
+---
+sidebar_position: 0
+---
+
+# Overview
+
+
+## Necessity of an upgrade
+
+After running multiple iterations of our incentivized testnets, inclding the latest version that engaged 5000+ Snapshotter Lite nodes, we gathered valuable metrics and data on the performance of the initial approach of our protocol.
+
+### Protocol State Bloat
+
+The snapshot submissions were so far resolved on a Solidity smart contract deployed on the Prost chain that acts as the consensus and reward mechanism for peers participating in the protocol. 
+
+This causes a state bloat of submissions for base snapshots for thousands of project IDs across 5000+ nodes, for every epoch. For a real time data market that updates every second, that is 86400 epochs in a day. Once the consensus on these snapshots have been validated, the data included in them becomes part of the higher order, composed datapoints and can be safely discarded from the chain state.
+
+:::info
+Learn more:
+
+* [Protocol State](/docs/Protocol/Specifications/protocol-state.md)
+* [Snapshot Generation](/docs/Protocol/Specifications/Snapshotter/snapshot-build.md)
+* [Introduction to Snapshotting consensus](/docs/build-with-powerloom/snapshotter-node/introduction)
+* [Accessing datasets on which consensus has been reached](/docs/build-with-powerloom/snapshotter-node/data)
+:::
+
+### Relayer transaction and chain infrastructure failures
+
+The above approach required the snapshotter peers to send out their submissions to the contract as an EIP-712 signed message object. This mechanism allowed us to trustlessly operate an infrastructure of relayers that would receive these submissions as HTTP POST requests and ultimately commit to the chain as EVM transactions. But at a scale of operation that exceeded 1 million snapshot submission transactions per day, this would often result in a significant percentage (>5%) of transactions being dropped along with other operational issues with maintaining the chain at a continuously running state.
+
+### Composed snapshots finality
+
+A combination of the above couple of issues also affects the [composability principle](/docs/Protocol/data-composition.md) of building higher order snapshots. Dropped transactions and increased delays in confirmation times of snapshot submissions ultimately affect the aggregate data points that power complex use cases like the [Uniswap V2 dashboard](/docs/build-with-powerloom/use-cases/existing-implementations/uniswapv2-dashboard/) and [Aave V3 dashboard](/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/).
+
+
+## Upgraded workflow
+
+The next iteration of the Powerloom Testnet and the upcoming Mainnet Alpha feature new components that are:
+
+* more accessible geographically
+* scalable
+* contribute to faster finality of composed snapshots
+
+![new architecture workflow](/images/new_architecture_workflow.png)
+
+It notably features the following components that make this possible.
+
+### Local collector: Snapshotter node
+
+The snapshotter node replaces the [payload commit service](/docs/Protocol/Specifications/Snapshotter/payload-commit-service.md) component with a faster grpc based service referred to as local collector. As the name suggests, it acts as a collection point for a stream of large number of [snapshots being built out](/docs/Protocol/Specifications/Snapshotter/snapshot-build.md) by the compute modules as specified in its configuration.
+
+It is responsible for reliably submitting all the generated snapshots to the sequencers via circuit relay nodes.
+
+### Sequencer
+
+The sequencer is a vital component in the upgraded workflow of the protocol. It batches submissions from multiple snapshotters within the active time window of an epoch, uploads them to IPFS and anchors the proof for the same on the protocol state contract. 
+
+* [Read more: Offchain sequencer](./sequencer.md)
+
+### libp2p circuit relay nodes
+
+The snapshot collector is ultimately able to submit snapshots to the sequencer component referenced above thanks to nodes that implement the [libp2p circuit relay protocol](https://docs.libp2p.io/concepts/nat/circuit-relay/).
+
+* [Read more: Circuit relay nodes](./relay.md)
+
+### Validator
+
+The upgraded protocol now supports validation on snapshot batches as submitted by the sequencer. The validators submit attestations for the fact that finalized snapshots across projects as reported by the sequencer are correct.
+
+* [Read more: Validator](./validator.md)

docs/Protocol/Protocol_v2/relay.md

@@ -0,0 +1,52 @@
+---
+sidebar_position: 2
+---
+
+# Relayers
+
+## Recap
+
+Powerloom Protocol V1 that powered the Incentivized testnets and the Lite Node testnet were able to submit their snapshots to the protocol state in the following manner: 
+
+1. The nodes submit their snapshots over HTTP to the relayers as EIP-712 signed message objects
+2. The relayers submit each node's submission against each project ID as [individual smart contract transactions to the method `submitSnapshot()`](/docs/Protocol/Specifications/protocol-state.md#function-submitsnapshotstring-memory-snapshotcid-uint256-epochid-string-memory-projectid-request-calldata-request-bytes-calldata-signature-public)
+
+## What's new in V2
+
+V2 relayers make use of the circuit relay transport protocol as implemented by the `libp2p` project. Briefly stated, this has two major advantages:
+
+* Faster, lightweight connections 
+  * Avoids overheads associated with HTTP connections: connection management, establishment, pooling, message payload size
+* More geographically available and censorship resistant
+
+To quote from the docs on circuit relays,
+
+```
+When a peer isn’t able to listen on a public address, it can dial out to a relay peer,
+which will keep a long-lived connection open. 
+
+Other peers will be able to dial through the relay peer using a 
+`p2p-circuit` address, which will forward traffic to its destination.
+```
+
+:::info
+[libp2p docs: Circuit Relay](https://docs.libp2p.io/concepts/nat/circuit-relay/)
+:::
+
+## Connection overview
+
+As stated in the Protocol V2 overview, the circuit relay nodes ultimately allow the snapshotter nodes to connect and submit to the sequencer node. This is intermediated further by the [Rendezvous protocol](https://docs.libp2p.io/concepts/discovery-routing/rendezvous/).
+
+:::note
+Rendezvous points are typically nodes that are well-connected and stable in a network and can handle large amounts of traffic and data. They serve as a hub for nodes to discover.
+:::
+
+The setup for the nodes includes `env` details that encapsulate sequencer ID(s), and the rendezvous points which intermediate and facilitate the connection between snapshotters, relayers and sequencer(s).
+
+## Final destination: Sequencer
+
+The circuit relay nodes ultimately allow Snapshotters on the Powerloom network, located anywhere in the world, to submit their snapshots to the sequencer.
+
+![snapshotter to sequencer via relay network](/images/SequencerRelayNetwork.png)
+
+In future upgrades of the protocol, we will begin supporting a decentralized network of sequencers with its own incentive and reward structure.

docs/Protocol/Protocol_v2/sequencer.md

@@ -0,0 +1,54 @@
+---
+sidebar_position: 1
+---
+# Sequencer
+
+![Powerloom Protocol V2 sequencer](/images/Sequencer.png)
+
+The sequencer has the following primary functions:
+
+1. **Snapshot Collection:** Receive snapshot submissions per epoch across project IDs from all participating snapshotters in a data market
+2. **Snapshot Finalization:** Finalize snapshot submission for each project ID according to consensus rules
+3. **Batch Upload Snapshot Submissions:** Create multiple batches of datasets of snapshot submissions and finalized CIDs across project IDs within an epoch.
+4. **Anchor Proof of Batch:** Upload these batches to IPFS or a decentralized storage layer. Commit the storage identifier to the protocol state smart contract for validators and other peers to reference for subsequent workflows according to their roles.
+
+
+## Snapshot collection and finalization
+
+The sequencer maintains an internal clock corresponding to the release and end of the submission window for an epoch. The collector component (not to be confused with the local collector component within a Snapshotter node) receives snapshot submissions from snapshotter peers over a libp2p circuit relay node.
+
+It also performs important verification checks:
+
+* identity verification by referencing an EIP-712 signature field contained within the submission payloads against the snapshotter identities allowed for a data market on the protocol state smart contract
+* submission window verification check by referencing the protocol state head block number as attached in the header of the submission sent by a snapshotter peer
+
+![Collector libp2p circuit relay connection](/images/Collector-relay.png)
+
+As the submission window for an epoch closes, the finalized submission for each project ID is decided upon by the sequencer according to rules of consensus, out of all the submissions sent out by participating snapshotter peers.
+
+## Batch upload and anchor proof
+
+Multiple such finalizations are decided upon all project IDs within a data market and they are batched together in multiple datasets, if required, at the end of every epoch's submission window. These datasets are then uploaded to IPFS or a decentralized storage layer, and references to their storage (for eg, a content identifier aka CID) are committed to the protocol state smart contract.
+
+The finalized snapshot submissions for all corresponding project IDs are placed in an 'intermediate' finalized state on the protocol state. They can be considered to be trusted, but are awaiting attestation from validators that participate in the protocol.
+
+:::info
+Read more on how validators submit attestations on intermediate finalized states
+:::
+
+
+Batches contain submission details for multiple project IDs with the following information per project:
+
+* Submission objects from snapshotter peers that contain their respectively signed EIP-712 objects
+* Finalized snapshot as reported by sequencer 
+
+![Batch submission section of protocol V2 workflow](/images/protov2-submission-batching.png)
+
+### Merkle tree of submission IDs
+
+An important feature of each batch is that the dataset against it contains the root hash of a Merkle tree constructed out of the submission IDs sent out by the snapshotter peers. Utilizing this, snapshotter peers can verify whether their submissions were included in batches committed to the protocol state smart contract for an epoch.
+
+The submission IDs contained in a batch are added to the leaves of the Merkle tree as determined by the lexicographical ordering of their corresponding project IDs. 
+
+Each batch's Merkle tree builds on the tree from the preceding batch within the same epoch.
+