chore: update build-with-powerloom (introduction, snapshotter node se…

…ctions)

docs/build-with-powerloom/introduction.md

@@ -2,53 +2,19 @@
 sidebar_position: 0
 ---
 
-# Snapshotter Peer Overview
+# Introduction
 
-The Snapshotter Peer is designed with a modular and highly configurable architecture, allowing for easy customization and seamless integration with a diverse set of data markets.
+Powerloom protocol is designed to be decentralized from the ground up. That means everything from data markets implementations to data relevance, everything is supposed to be controlled by market economics. This is a very different approach from the current centralized solutions where the data is controlled by a single entity.
 
-![Submodule architecture of snapshotter](/images/submodule_architecture.png)
+The overall architecture of Powerloom protocol is as follows:
+![Overall Architecture](/images/OverallArchitecture.png)
 
+One of the most important parts of the Powerloom protocol is the Snapshotter peer. Snapshotter is a node that is responsible for reading data from various blockchains and sending it to the Powerloom consensus protocol for validation. Anyone can run a snapshotter node and join the Powerloom network.
 
-## Snapshotter Core
+Building a new use case can often be very tricky because of the lack of documentation and examples. This section aims to provide a comprehensive walkthrough on how to build a new use case on top of Powerloom protocol.
 
-This foundational component defines all the essential interfaces and handles a wide range of tasks, from listening to epoch release events to distributing tasks and managing snapshot submissions. Read more about it in the detailed section on its [components](/docs/protocol/specifications/snapshotter/components).
+We aim to build a community owned repository of wide use case implementation with anyone being able to contibute to it. All it needs is couple of `.env` variables linking to the repository with use case and configuration files.
 
+Eventually we aim to build a marketplace where anyone can deploy their use case and start earning from it.
 
-## Data market specific
-
-Use case specific logic of generating snapshots as well as other configuration are available as [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) which can be pulled in as desired. Presently you can find diverse use cases that serve customized datapoints and track activity for the following across multiple data source chains like Ethereum Mainnet, Polygon zkEVM, Polygon PoS etc.
-
-* DEXs like Uniswap v2, Quickswap
-* Bridges like bungee.exchange, Owlto finance
-
-The architecture has been designed to facilitate the seamless interchange of configuration and modules. Adapting the system to different use cases is as straightforward as changing a Git branch.
-
-You can observe the corresponding branches within [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/) and [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/) repos:
-
-#### Snapshotter Computes
-* `eth_uniswapv2`: Pooler implementation for Uniswap v2 dashboard
-* `eth_uniswapv2_lite`: Pooler implementation for Uniswap v2 dashboard but without calculation of complex aggregate data points from scratch.
-* `zkevm_quests`: Implementation for Quests on Polygon zkEVM
-
-#### Snapshotter Configs
-* `eth_uniswapv2_5_pairs`: Pooler lite mode config with only 5 chosen uniswap v2 pair contracts
-* `eth_uniswapv2`: Pooler config with only 180 uniswap v2 pair contracts
-* `zkevm_quests`: Config for Quests on Polygon zkEVM
-
-### Configuration Files
-
-Configuration files, located in the `/config` directory are linked to [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/) repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings.
-
-
-### Compute Modules
-
-The heart of the system resides in the `snapshotter/modules` directory that's linked to [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/), where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types.
-
-## Building your own use case
-Working on a new use case is as simple as writing a new compute module and adding a new configuration file. The instructions to get started in `build-dev.sh` mode are available in the [Deploy repo](https://github.com/PowerLoom/deploy/tree/eth_india).
-
-# Useful links
-
-* [Snapshot generation specifications](/docs/protocol/specifications/snapshotter/snapshot-build)
-* [Data markets and sources](/docs/protocol/data-sources)
-* [Composition of snapshots and higher order datapoints](/docs/protocol/data-composition)
+Let's dive in Snapshotter node and see how it all works!

docs/build-with-powerloom/snapshotter-node/_category_.json

@@ -1,6 +1,6 @@
 {
     "label": "Snapshotter Node",
-    "position": 0,
+    "position": 1,
     "link": {
       "type": "generated-index",
       "description": "Dive deep into how snapshotter peers transform the powerloom network"

docs/build-with-powerloom/snapshotter-node/architecture.md

@@ -2,4 +2,52 @@
 sidebar_position: 1
 ---
 
-## Architecture
+# Architecture
+The Snapshotter Peer is designed with a modular and highly configurable architecture, allowing for easy customization and seamless integration with a diverse set of data markets.
+
+![Submodule architecture of snapshotter](/images/submodule_architecture.png)
+
+
+## Snapshotter Core
+
+This foundational component defines all the essential interfaces and handles a wide range of tasks, from listening to epoch release events to distributing tasks and managing snapshot submissions. Read more about it in the detailed section on its [components](/docs/protocol/specifications/snapshotter/components).
+
+
+## Data market specific
+
+Use case specific logic of generating snapshots as well as other configuration are available as [Git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) which can be pulled in as desired. Presently you can find diverse use cases that serve customized datapoints and track activity for the following across multiple data source chains like Ethereum Mainnet, Polygon zkEVM, Polygon PoS etc.
+
+* DEXs like Uniswap v2, Quickswap
+* Bridges like bungee.exchange, Owlto finance
+
+The architecture has been designed to facilitate the seamless interchange of configuration and modules. Adapting the system to different use cases is as straightforward as changing a Git branch.
+
+You can observe the corresponding branches within [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/) and [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/) repos:
+
+#### Snapshotter Computes
+* `eth_uniswapv2`: Pooler implementation for Uniswap v2 dashboard
+* `eth_uniswapv2_lite`: Pooler implementation for Uniswap v2 dashboard but without calculation of complex aggregate data points from scratch.
+* `zkevm_quests`: Implementation for Quests on Polygon zkEVM
+
+#### Snapshotter Configs
+* `eth_uniswapv2_5_pairs`: Pooler lite mode config with only 5 chosen uniswap v2 pair contracts
+* `eth_uniswapv2`: Pooler config with only 180 uniswap v2 pair contracts
+* `zkevm_quests`: Config for Quests on Polygon zkEVM
+
+### Configuration Files
+
+Configuration files, located in the `/config` directory are linked to [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/) repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings.
+
+
+### Compute Modules
+
+The heart of the system resides in the `snapshotter/modules` directory that's linked to [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/), where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types.
+
+## Building your own use case
+Working on a new use case is as simple as writing a new compute module and adding a new configuration file. The instructions to get started in `build-dev.sh` mode are available in the [Deploy repo](https://github.com/PowerLoom/deploy/tree/eth_india).
+
+# Useful links
+
+* [Snapshot generation specifications](/docs/protocol/specifications/snapshotter/snapshot-build)
+* [Data markets and sources](/docs/protocol/data-sources)
+* [Composition of snapshots and higher order datapoints](/docs/protocol/data-composition)

docs/build-with-powerloom/snapshotter-node/core-api/_category_.json

@@ -1,8 +1,8 @@
 {
     "label": "Snapshotter Core API",
-    "position": 2,
+    "position": 4,
     "link": {
       "type": "generated-index",
-      "description": "Dive deep into how snapshotter peers transform the powerloom network"
+      "description": "Snapshotter Core API"
     }
   }

docs/build-with-powerloom/snapshotter-node/core-api/index.md

@@ -0,0 +1,9 @@
+---
+sidebar_position: 0
+---
+
+# Introduction
+
+Core API facilitates access to the finalized protocol state on the smart contract running on the anchor chain. Although it is possible to directly query the smart contract, it is not recommended as it is expensive and slow. Instead, you can use the Core API to access the finalized state. Every snapshotter node runs a Core API service that can be accessed by other services or applications.
+
+We have hosted snapshotter nodes for UniswapV2 and Zkevm Quests on the PowerLoom Protocol's testnet. You can interact with those endpoints directly from the next couple of sections.

docs/build-with-powerloom/snapshotter-node/core-api/snapshotter-core-api.info.mdx

@@ -2,7 +2,7 @@
 id: snapshotter-core-api
 title: "Snapshotter Core API"
 description: ""
-sidebar_label: Introduction
+sidebar_label: Auth Info
 sidebar_position: 0
 hide_title: true
 custom_edit_url: null

docs/build-with-powerloom/snapshotter-node/data.md

@@ -0,0 +1,89 @@
+---
+sidebar_position: 3
+---
+
+# All about Data
+So far we've talked a lot of verifiablity and decentralization of data. Let's dive into the details of how data is stored and how it can be verified and accessed.
+
+## Data Storage
+Data is stored in a decentralized manner using IPFS. IPFS is a peer-to-peer network for storing and sharing data in a distributed file system. It aims to replace HTTP and build a better web for all of us. IPFS is a peer-to-peer distributed file system that seeks to connect all computing devices with the same system of files. In some ways, IPFS is similar to the Web, but IPFS could be seen as a single BitTorrent swarm, exchanging objects within one Git repository. 
+
+In other words, IPFS provides a high throughput content-addressed block storage model, with content-addressed hyperlinks. This forms a generalized Merkle DAG, a data structure upon which one can build versioned file systems, blockchains, and even a Permanent Web. IPFS combines a distributed hashtable, an incentivized block exchange, and a self-certifying namespace. IPFS has no single point of failure, and nodes do not need to trust each other.
+
+Every file and every block of data on the network has a unique hash associated with it. This hash is used to identify the data and is used to retrieve it from the network. This hash is called a CID (Content Identifier).
+
+## Fetching CID and Verifying it On-Chain
+![API Calls for the a certain CID](/images/uniswap-inspect.png)
+
+Powerloom Uniswap V2 dashboard (https://uniswapv2.powerloom.io/) for example, connects to a snapshotter node hosted by the Powerloom Foundation. 
+The frontend interacts with [Snapshotter Core API](/docs/category/snapshotter-core-api) to access the data. Internally, the Core API interacts with the Protocol State Contract deployed on Powerloom Prost chain to fetch the data.
+
+### Verifying Data
+To verify the data, all you need is Prost chain RPC Url and the address of the deployed protocol state contract. You can find these details in the `env.example` file in the Powerloom [deploy](https://github.com/PowerLoom/deploy/) repository.
+
+#### Adding the Chain to the Wallet
+
+Once you have these details:
+
+1. Open MetaMask or any web3 plugin in your browser.
+2. Go to the network selection area.
+3. Choose to add a new network.
+4. Enter the RPC URL and contract address you found in the `env.example` file.
+
+This will allow you to connect to the Powerloom Prost chain. Metamask example is shown in the screenshot below.
+
+Note: The RPC URL and the contract address in the screenshot below are just for reference. You need to use the ones from the `env.example` file.
+
+![Chain-Details](/images/Chain-details.png)
+
+#### Accessing the ABI of the protocol state Contract
+
+To access and utilize the ABI of the protocol state contract from the Powerloom project, follow these steps:
+
+1. Visit the Powerloom 'pooler' repository on GitHub at this URL: [Powerloom pooler repository - ProtocolContract.json](https://github.com/PowerLoom/pooler/blob/main/snapshotter/static/abis/ProtocolContract.json).
+2. Locate the `ProtocolContract.json` file.
+3. Copy the contents of the file.
+4. Open the Remix IDE.
+5. Create a new file with a `.abi` extension.
+6. Paste the contents of the `ProtocolContract.json` file into this new file.
+7. Save the file in Remix.
+
+This process will allow you to have the protocol state contract’s ABI available in the Remix editor for further development or interaction.
+![ABI Interacting](/images/abi-interaction.png)
+
+**Verifying datasets**
+
+To verify data for any Project ID, follow these steps:
+
+1. Identify the last finalized epoch associated with the project ID: For example, `aggregate_24h_top_pairs_lite:10ecae2f52160690abffff26efeb45568e5d67ea0bc7d4485d9ffb10ef437f33:UNISWAPV2`.
+2. Retrieve the finalized CID (Content Identifier) for this epoch.
+3. Access the dataset linked to this CID. You can do this either through your local IPFS node or by using a public IPFS gateway.(Public gateway link: https://ipfs.io/ipfs/{IPFS-CID})
+
+The exact logic to retrieve the CID from the protocol state contract can easily be extracted from the [Snapshotter Core API](/docs/category/snapshotter-core-api) codebase.
+
+Here's the code to find last finalized epoch for a project ID:
+```python reference
+https://github.com/PowerLoom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/core_api.py#L248-L339
+```
+
+Here's the code to fetch the data for a given epoch ID and project ID:
+```python reference
+https://github.com/PowerLoom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/core_api.py#L343-L404
+```
+
+All you need to do is call the relevant functions in the loaded Protocol state contract ABI in Remix IDE to fetch the data.
+
+
+:::tip
+Sometimes, the IPFS gateway can be overloaded or data might not be peer-to-peer replicated and it can result in not giving out an output. In that case, you can tunnel into your node and verify the data. 
+
+```bash
+
+```bash
+ssh -nNTv -L 5001:127.0.0.1:5001 root@yourServerIP
+```
+
+```bash
+ipfs cat bafkreifc33xiuqaf5nxqad2syn7j5x27j2pujzr7hhsxgtsr7zor4svqie --api /ip4/127.0.0.1/tcp/5001
+```
+:::

docs/build-with-powerloom/snapshotter-node/getting-started.md

@@ -0,0 +1,27 @@
+---
+sidebar_position: 2
+---
+# Getting Started
+
+The snapshotter is part of a distributed system with multiple moving parts. The easiest way to get started is by using the Docker-based setup. This setup is recommended for most users.
+
+## Requirements
+```markdown reference
+https://github.com/PowerLoom/deploy/blob/7708906fc606df2eff84642fe54d2c47bd9137ab/README.md#L6-L12
+```
+
+## For Snapshotters
+```markdown reference
+https://github.com/PowerLoom/deploy/blob/60c7be94160b5d9f6e89a41ac909cf41c1fdcb24/README.md#L18-L53
+```
+
+## For Code Contributors
+```markdown reference
+https://github.com/PowerLoom/deploy/blob/7708906fc606df2eff84642fe54d2c47bd9137ab/README.md#L15-L74
+```
+
+```markdown reference
+https://github.com/PowerLoom/deploy/blob/7708906fc606df2eff84642fe54d2c47bd9137ab/README.md#L109-L115
+```
+
+**Note** - RPC usage is highly use-case specific. If your use case is complicated and needs to make a lot of RPC calls, it is recommended to run your own RPC node instead of using third-party RPC services as it can be expensive.

docs/build-with-powerloom/snapshotter-node/introduction.md

@@ -0,0 +1,43 @@
+---
+sidebar_position: 0
+---
+
+# Introduction
+
+![Snapshotter workflow](/images/snapshotter_workflow.png)
+
+A snapshotter peer as part of Powerloom Protocol does exactly what the name suggests:  It synchronizes with other snapshotter peers over a smart contract running on Powerloom Prost chain. It follows an architecture that is driven by state transitions which makes it easy to understand and modify.
+
+Because of its decentralized nature, the snapshotter specification and its implementations share some powerful features that can adapt to your specific information requirements on blockchain applications:
+
+* Each data point is calculated, updated, and synchronized with other snapshotter peers participating in the network
+* synchronization of data points is defined as a function of an epoch ID(identifier) where epoch refers to an equally spaced collection of blocks on the data source blockchain (for eg, Ethereum Mainnet/Polygon Mainnet/Polygon Testnet -- Mumbai). This simplifies the building of use cases that are stateful (i.e. can be accessed according to their state at a given height of the data source chain), synchronized, and depend on reliable data. For example,
+    * dashboards by offering higher-order aggregate datapoints
+    * trading strategies and bots
+* a snapshotter peer can load past epochs, indexes, and aggregates from a decentralized state and have access to a rich history of data
+    * all the datasets are decentralized on IPFS/Filecoin
+    * the power of these decentralized storage networks can be leveraged fully by applying the [principle of composability](/docs/protocol/data-composition)
+
+
+Powerloom relies on snapshotter peers to generate consensus backed data instead of relying on centralized oracles. This approach helps in mitigating issues like single points of failure, outdated data, or dependence on particular data providers. The decentralized, peer-to-peer structure of Powerloom ensures consistent availability and reliability of consensus backed data.
+
+![Powerloom Workflow](/images/introduction-image.png)
+
+
+## Specifications and moving parts
+
+Snapshotter is a crucial part of our powerloom protocol. Dive deep into the technical aspects of snapshotter [here](/docs/category/snapshotter)
+
+## Joining a data market
+
+Continue along within this category to explore the different testnet based data markets that you can join as a snapshotter.
+
+* [Joining a network](/docs/category/joining-networks)
+
+
+:::note
+
+RPC usage is highly use-case specific. If your use case is complicated and needs to make a lot of RPC calls, it is recommended to run your own RPC node instead of using third-party RPC services as it can be expensive.
+
+:::
+