Merge pull request #133 from covalenthq/develop

Rudder-DTM-RC:v0.2.11
covalenthq · Jun 19, 2023 · 4423b5d · 4423b5d
2 parents 4cceea7 + 1a6ad4d
commit 4423b5d
Show file tree

Hide file tree

Showing 14 changed files with 64 additions and 42 deletions.
diff --git a/README.md b/README.md
diff --git a/docs/ARCH.md b/docs/ARCH.md
@@ -10,11 +10,11 @@
   - [Pipeline Journal](#pipeline-journal)
   - [Pipeline Telemetry](#pipeline-telemetry)
 
-![Rudder Pipeline](./pipeline.jpg)
+![Rudder Pipeline](./pipeline-white.png)
 
-The happy path for `rudder` (the refiner) application in the Covalent Network is made up of actor processes spawned through many [Gen Servers](https://elixir-lang.org/getting-started/mix-otp/genserver.html) processes that are loosely coupled, here some maintain state and some don't.
+The happy path for the `rudder` (the refiner) application in the Covalent Network is made up of actor processes spawned through many [Gen Servers](https://elixir-lang.org/getting-started/mix-otp/genserver.html) processes that are loosely coupled, here some maintain state, and some don't.
 
-The children processes can be called upon to fulfill responsibilities at different sections in the refinement/transformation process pipeline - under one umbrella [Dynamic Supervisor](https://elixir-lang.org/getting-started/mix-otp/dynamic-supervisor.html), that can bring them back up in case of a failure to continue a given pipeline operation.
+The children processes can be called upon to fulfill responsibilities at different sections in the refinement/transformation process pipeline - under one umbrella [Dynamic Supervisor](https://elixir-lang.org/getting-started/mix-otp/dynamic-supervisor.html) that can bring them back up in case of a failure to continue a given pipeline operation.
 
 ![Rudder Supervisor](./supervisor.png)
 
@@ -29,16 +29,15 @@ There are currently 8 main components to the refiner.
   7. Pipeline Journal
   8. Pipeline Telemetry
 
-
-
 ## <span id="rudder_arch_listen">Block Specimen Event Listener</span>
 
 The block specimen event listener is the first to start in the rudder/refiner pipeline by listening to events happening in the [proof-chain contract](https://github.com/covalenthq/bsp-staking). The events refiner cares about relate to the finalized block specimens and the finalized block results.
 
-For the former for all block specimen proofs that have been submitted and achieved consensus are ready for being transformed in block results.
+All block specimen proofs that have been submitted and achieved consensus are ready to be transformed into block results.
 
 Start the listener.
 
+
 ```elixir
 iex -S mix
 
@@ -62,7 +61,7 @@ tail -f logs/log.log
 
 ## <span id="rudder_arch_encode_decode">Block Specimen Encoder Decoder</span>
 
-Once a block specimen that has been finalized has been received, rudder extracts the specimen directly async in the pipeline process, spawning a block specimen decode process for each specimen separately using AVRO client library `avrora`.
+Once a block specimen that has been finalized has been received, the rudder extracts the specimen directly async in the pipeline process, spawning a block specimen decode process for each specimen separately using AVRO client library `avrora`.
 
 It carries out the following steps -
 
@@ -115,11 +114,11 @@ iex(2)> Rudder.Avro.BlockSpecimenDecoder.decode_dir("test-data/*")
 
 ## <span id="rudder_arch_processor">Block Specimen Processor</span>
 
-Next the block specimen processor available as an http server with `export EVM_SERVER_URL="http://127.0.0.1:3002"`. This takes the `block_id` and `block_specimen` json object and provides the block result. The stateless transition tool needed to run the specimen, is written in `golang`, which is invoked via the http server outside of rudder.
+Next, the block specimen processor is available as an http server with `export EVM_SERVER_URL="http://127.0.0.1:3002"`. This takes the `block_id` and `block_specimen` json object and provides the block result. The stateless transition tool needed to run the specimen is written in `golang`, which is invoked via the http server outside of the rudder.
 
-In an earlier version of rudder server the server was originally a (golang) binary plugin to the rudder application and executed with the block specimen inputs in a `:porcelain` app within a shell process in erlang, but then moved out due to performance and consistency considerations.
+In an earlier version of the rudder, the server was originally a (golang) binary plugin to the rudder application and executed with the block specimen inputs in a `:porcelain` app within a shell process in erlang, but then moved out due to performance and consistency considerations.
 
-Below is an example of submitting a avro encoded block specimen binary to the specimen processor.
+Below is an example of submitting an avro encoded block specimen binary to the specimen processor.
 
 ```elixir
 iex(1)> replica_fp="test-data/1-15127602-replica-0xce9ed851812286e05cd34684c9ce3836ea62ebbfc3764c8d8a131f0fd054ca35"
@@ -145,7 +144,7 @@ That will lead to the corresponding logs:
 
 ## <span id="rudder_arch_uploader">Block Result Uploader</span>
 
-Once the block results have been produced they need to be proved and uploaded. This ideally happens atomically for rudder.
+Once the block results have been produced they need to be proved and uploaded. This ideally happens atomically for the rudder.
 
 Below is an example of how to interact with block result uploader that speaks to `ipfs-pinner` available with `export IPFS_PINNER_URL="http://127.0.0.1:3001"`. The file is directly uploaded to IPFS using the wrapped local IPFS node.
 
@@ -180,7 +179,7 @@ rudder        | 08:53:06.414 [info] 1:525D191D6492F1E0928d4e816c29778c proof sub
 
 ## <span id="rudder_arch_ipfs_interact">IPFS Interactor</span>
 
-Underlying the block result uploader is the IPFS interactor module that allows refiner to interact with IPFS cids, by listening for them and uploading them. Below is an example to fetch or discover a block specimen using its uploaded `cid` collected by listening to the log event of a block specimen proof submission.
+Underlying the block result uploader is the IPFS interactor module that allows refiner to interact with IPFS cids by listening for them and uploading them. Below is an example of fetching or discovering a block specimen using its uploaded `cid` collected by listening to the log event of a block specimen proof submission.
 
 ```elixir
 iex(1)> urls=["ipfs://bafybeifo5o7zatnudfyvixkziy5aj4fhikv5nq3pbizpgwdcz4fqwarhgu"]
@@ -267,9 +266,9 @@ Rudder keeps track of all the queue items that are in the following states of pr
 
 ## <span id="rudder_arch_journal">Pipeline Telemetry</span>
 
-Rudder records various metrics like `counter`, `lastvalue`, `sum` and `summary` during a pipeline process within each module and stores them in an ETF table by modules. This can then be used to understand how the processes have been progressing and where the performance bottlenecks may lay.
+Rudder records metrics like `counter`, `lastvalue`, `sum`, and `summary` during a pipeline process within each module and stores them in an ETF table by modules. This can then be used to understand how the processes have been progressing and where the performance bottlenecks may lay.
 
-The logs that pertain to the telemetry and performance of rudder can be seen as follows from fetching start (specimen event) to finish (pipeline success).
+The logs that pertain to the telemetry and performance of the rudder can be seen as follows from fetching start (specimen event) to finish (pipeline success).
 
 ```elixir
 rudder       | [info] starting event listener

diff --git a/docs/development.md → docs/DEVELOPMENT.md b/docs/development.md → docs/DEVELOPMENT.md
diff --git a/docs/metrics.md → docs/METRICS.md b/docs/metrics.md → docs/METRICS.md
diff --git a/docs/README.md b/docs/README.md
@@ -0,0 +1,19 @@
+# Index
+
+## Useful Documents
+
+1. [Full Architecture Doc](./ARCH.md)
+1. [Telemetry & Metrics Doc](./METRICS.md)
+1. [Contributing Doc](./CONTRIBUTING.md)
+1. [Development Doc](./DEVELOPMENT.md)
+
+## List of Diagrams
+
+1. Phase 2 Refiner![Phase 2 Refiner](./phase-2.png)
+1. Covalent Network Layers![Covalent Network Layers](./network-layers.png)
+1. Full Architecture of Covalent Network![Full Architecture of Covalent Network](./arch-white.png)
+1. Rudder/Refiner Components![Rudder/Refiner Components](./components.png)
+1. Rudder/Refiner Pipeline![Rudder/Refiner Pipeline](./pipeline-white.png)
+1. Rudder Dynamic Supervisor![Rudder Dynamic Supervisor](./supervisor.png)
+1. Prometheus Dashboard![Prometheus Dashboard](./prometheus.png)
+1. Grafana Dashboard![Grafana Dashboard](./dashboard.png)
diff --git a/docs/arch-light.jpg b/docs/arch-light.jpg
diff --git a/docs/arch-white.png b/docs/arch-white.png
diff --git a/docs/components.png b/docs/components.png
diff --git a/docs/network-layers.png b/docs/network-layers.png
diff --git a/docs/phase-2.png b/docs/phase-2.png
diff --git a/docs/pipeline.jpg → docs/pipeline-blue.jpg b/docs/pipeline.jpg → docs/pipeline-blue.jpg
diff --git a/docs/pipeline-white.png b/docs/pipeline-white.png
diff --git a/mix.exs b/mix.exs
@@ -4,7 +4,7 @@ defmodule Rudder.MixProject do
   def project do
     [
       app: :rudder,
-      version: "0.2.10",
+      version: "0.2.11",
       elixir: "~> 1.14.3",
       start_permanent: Mix.env() == :prod,
       deps: deps()

diff --git a/mix.lock b/mix.lock
@@ -27,7 +27,7 @@
   "file_system": {:hex, :file_system, "0.2.10", "fb082005a9cd1711c05b5248710f8826b02d7d1784e7c3451f9c1231d4fc162d", [:mix], [], "hexpm", "41195edbfb562a593726eda3b3e8b103a309b733ad25f3d642ba49696bf715dc"},
   "finch": {:hex, :finch, "0.16.0", "40733f02c89f94a112518071c0a91fe86069560f5dbdb39f9150042f44dcfb1a", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: false]}, {:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 1.3", [hex: :mint, repo: "hexpm", optional: false]}, {:nimble_options, "~> 0.4 or ~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_pool, "~> 0.2.6 or ~> 1.0", [hex: :nimble_pool, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "f660174c4d519e5fec629016054d60edd822cdfe2b7270836739ac2f97735ec5"},
   "gen_stage": {:hex, :gen_stage, "1.2.1", "19d8b5e9a5996d813b8245338a28246307fd8b9c99d1237de199d21efc4c76a1", [:mix], [], "hexpm", "83e8be657fa05b992ffa6ac1e3af6d57aa50aace8f691fcf696ff02f8335b001"},
-  "gettext": {:hex, :gettext, "0.22.1", "e7942988383c3d9eed4bdc22fc63e712b655ae94a672a27e4900e3d4a2c43581", [:mix], [{:expo, "~> 0.4.0", [hex: :expo, repo: "hexpm", optional: false]}], "hexpm", "ad105b8dab668ee3f90c0d3d94ba75e9aead27a62495c101d94f2657a190ac5d"},
+  "gettext": {:hex, :gettext, "0.22.2", "6bfca374de34ecc913a28ba391ca184d88d77810a3e427afa8454a71a51341ac", [:mix], [{:expo, "~> 0.4.0", [hex: :expo, repo: "hexpm", optional: false]}], "hexpm", "8a2d389673aea82d7eae387e6a2ccc12660610080ae7beb19452cfdc1ec30f60"},
   "hackney": {:hex, :hackney, "1.18.1", "f48bf88f521f2a229fc7bae88cf4f85adc9cd9bcf23b5dc8eb6a1788c662c4f6", [:rebar3], [{:certifi, "~> 2.9.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~> 6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~> 1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~> 1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.3.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~> 1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "a4ecdaff44297e9b5894ae499e9a070ea1888c84afdd1fd9b7b2bc384950128e"},
   "hpax": {:hex, :hpax, "0.1.2", "09a75600d9d8bbd064cdd741f21fc06fc1f4cf3d0fcc335e5aa19be1a7235c84", [:mix], [], "hexpm", "2c87843d5a23f5f16748ebe77969880e29809580efdaccd615cd3bed628a8c13"},
   "httpoison": {:hex, :httpoison, "1.8.2", "9eb9c63ae289296a544842ef816a85d881d4a31f518a0fec089aaa744beae290", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "2bb350d26972e30c96e2ca74a1aaf8293d61d0742ff17f01e0279fef11599921"},