Migrate from tzstats to tzpro (#685)

* migration from tzstats to tzpro * upgraded required package versions and applied black formatting * attempt: fix test dependencies, move verbose_logger_test into test folder * remove bad import module prefix 'src.' * fix requirements for python 3.7, fix module references in tests * Add tzpro api key in config * Update tzpro data * Fix enum compares * Currently failing tests: test_attempt_single_batch_tz test_attempt_single_batch_KT test_batch_payer_total_payout_amount test_simulate_single_operation test_failed_simulate_single_operation test_reset test_archive * Contributor/Reviewer: jdsika, Effort=6h, vkresch, Effort=8h --------- Signed-off-by: jdsika <carlo.van-driesten@vdl.digital> Co-authored-by: Viktor <viktor.kreschenski@gmail.com>
tezos-reward-distributor-organization · Nov 4, 2023 · c852b20 · c852b20
1 parent d4fd27a
commit c852b20
Show file tree

Hide file tree

Showing 95 changed files with 515 additions and 539 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -61,6 +61,8 @@ jobs:
         pip install -r requirements_developers.txt
 
     - name: Test with pytest
+      env:
+        TZPRO_API_KEY: ${{ secrets.TZPRO_API_KEY }} 
       run: pytest tests/
 
   build-documentation:

diff --git a/docs/configuration.rst b/docs/configuration.rst
@@ -174,7 +174,13 @@ Available configuration parameters are:
        tz1V9SpwXaGFiYdDfGJtWjA61EumAH3DwSyT: tz1fgX6oRWQb4HYHUT6eRjW8diNFrqjEfgq7        #(redirects payment from tz1V9S... to tz1fgX...)
        tz1RMmSzPSWPSSaKU193Voh4PosWSZx1C7Hs: Dexter                                      #(indicates address is a dexter pool; TRD will send rewards to pool members)
        mindelegation: TOE                                                                #(mindelegation will be shared with everyone)
-  
+
+**tzpro_api_key**
+  Generate a tzpro API key [here](https://tzpro.io/) if you want to use it as reward or block API.
+
+  Example::
+    tzpro_api_key: XXXXXXXXXX
+
 **plugins**
   Please consult the `plugins docs`_ for more details on the configuring the various plugins.
 

diff --git a/docs/index.rst b/docs/index.rst
@@ -10,7 +10,7 @@ PRIVACY : TEZOS REWARD DISTRIBUTOR COLLECTS ANONYMOUS STATISTICS. PLEASE READ OU
 What's TRD?
 ------------------------------------------------
 
-TRD is an open-source software for distributing staking rewards from bakers to delegators introduced in detail in this Medium article_. This is not a python script but a full scale application which can continuously run in the background as a Linux service. However it does not have to be used as a service, but it can also be used interactively. The tool convinces with its simplicity and yet leaves no configuration wish unfulfilled. Whether minimum delegation threshold, special fees for some delegators, or actual vs ideal rewards - the TRD covers just about all possible constellations. Furthermore, the tool supports complex payments, pays in batches, and provides three back ends for calculations: Tezos RPC, tzstats_ API and TzKT_ API. TRD is developed and tested extensively by the community and the source code which can be found in the following Github_ repo.
+TRD is an open-source software for distributing staking rewards from bakers to delegators introduced in detail in this Medium article_. This is not a python script but a full scale application which can continuously run in the background as a Linux service. However it does not have to be used as a service, but it can also be used interactively. The tool convinces with its simplicity and yet leaves no configuration wish unfulfilled. Whether minimum delegation threshold, special fees for some delegators, or actual vs ideal rewards - the TRD covers just about all possible constellations. Furthermore, the tool supports complex payments, pays in batches, and provides three back ends for calculations: Tezos RPC, tzpro_ API and TzKT_ API. TRD is developed and tested extensively by the community and the source code which can be found in the following Github_ repo.
 
 Who needs TRD?
 ------------------------------------------------
@@ -23,21 +23,19 @@ What else do you need for TRD?
 There are currently the following options to run TRD:
 
     a. If you want to use RPC (not public RPC) for the reward calculation, you need a Tezos archive node. 
-    b. If you want to use an provider (pRPC, tzstats, tzkt) for the reward calculation, but want to inject your own transactions, at least a Tezos rolling node is needed.
-    c. If you want to use an provider (pRPC, tzstats, tzkt) for the reward calculation and don't want to inject your own transactions, only the Tezos signer is needed.
+    b. If you want to use an provider (pRPC, TZPRO, tzkt) for the reward calculation, but want to inject your own transactions, at least a Tezos rolling node is needed.
+    c. If you want to use an provider (pRPC, TZPRO, tzkt) for the reward calculation and don't want to inject your own transactions, only the Tezos signer is needed.
 
 However, for all options the Tezos signer is needed.
 
 **Provider notes:**
 
-TZStats
+Blockwatch: TZPRO
 -----------
 
-The terms_ of tzstats note that a license is needed for the commercial use of their API:
+The terms_ of TZPRO note that an account and API key are needed for the use of the API. Please review the [pricing](https://tzpro.io/#pricing) information. For further help contact hello@blockwatch.cc for more information.
 
-    If you wish to use the Data in a manner that is primarily intended for or directed towards commercial advantage or monetary compensation (such use, “Commercial Use”), Blockwatch requires that you enter into a separate commercial license agreement. Entering into a separate commercial license allows us to protect Blockwatch’s investment in the Data and to maintain the integrity of the Data.
-
-    Please contact us at license@blockwatch.cc for more information about Commercial Uses of our Data.
+In order to use your API key in the application copy and rename the .env.example to .env and add the API key for TZPRO.
 
 TzKT
 -----------
@@ -52,11 +50,11 @@ With PR232_ the backend of the Tezos Reward Distributor can be optionally `Power
 
 .. _article : https://medium.com/@huseyinabanox/tezos-reward-distributor-e6588c4d27e7
 
-.. _tzstats : https://tzstats.com/
+.. _tzpro: https://tzpro.io/
 
 .. _TzKT : https://api.tzkt.io/
 
-.. _terms : https://tzstats.com/terms
+.. _terms : https://tzpro.io/terms
 
 .. _Github : https://github.com/tezos-reward-distributor-organization/tezos-reward-distributor
 

diff --git a/docs/run.rst b/docs/run.rst
@@ -41,11 +41,11 @@ Options
 ``-A --node_endpoint <node_url:port>``
     Node potentially with protocol prefix especially if TLS encryption is used. Default value: ``http://127.0.0.1:8732``. This is the main Tezos node used by the client for RPC queries and operation injections.
 
-``-P --reward_data_provider <rpc|prpc|tzstats|tzkt>``
-    Source that provides all needed data for reward calculations. Default value: ``tzkt`` (TzKT API). Set to ``rpc`` to use your own local node defined with the ``-A`` flag, (it must be an ARCHIVE node in this case). Set to ``prpc`` to use a public RPC node defined with the ``-Ap`` flag. An alternative for providing reward data is ``tzstats``, but pay attention for license in case of commercial use!
+``-P --reward_data_provider <rpc|prpc|tzpro|tzkt>``
+    Source that provides all needed data for reward calculations. Default value: ``tzkt`` (TzKT API). Set to ``rpc`` to use your own local node defined with the ``-A`` flag, (it must be an ARCHIVE node in this case). Set to ``prpc`` to use a public RPC node defined with the ``-Ap`` flag. An alternative for providing reward data is ``tzpro``, but an API key associated with your account needs to be provided in the .env file!
 
 ``-Ap --node_addr_public <url>``
-    Public node base URL. Default is ``https://rpc.tzstats.com``. This argument will only be used in case the reward provider is set to ``prpc``. This node will only be used to query reward data and delegator list. It must be an ARCHIVE node.
+    Public node base URL. Default is ``https://mainnet.smartpy.io``. This argument will only be used in case the reward provider is set to ``prpc``. This node will only be used to query reward data and delegator list. It must be an ARCHIVE node.
 
 ``-b --base_directory <path>``
     Directory for reports, configuration and logs. Default value: ``~/pymnt``.
@@ -131,13 +131,13 @@ Run in dry-run mode on MAINNET, make payouts only for cycle 300, for calculation
 
 ::
 
-    python3 src/main.py -C 300 -P prpc -Ap https://rpc.tzstats.com -M 3 -V -D
+    python3 src/main.py -C 300 -P prpc -Ap https://mainnet.smartpy.io -M 3 -V -D
 
-Run in dry-run mode on MAINNET, retry failed payouts only for cycle 300, for calculations use data provided by the TzStats API:
+Run in dry-run mode on MAINNET, retry failed payouts only for cycle 300, for calculations use data provided by the TZPRO API:
 
 ::
 
-    python3 src/main.py -C 300 -P tzstats -M 4 -V -D
+    python3 src/main.py -C 300 -P tzpro -M 4 -V -D
 
 For help, run:
 

diff --git a/examples/lambda_stats.py b/examples/lambda_stats.py
@@ -20,7 +20,6 @@
 
 
 def lambda_handler(event, context):
-
     getpost = event["requestContext"]["http"]["method"].lower()
 
     # check if POST
@@ -66,7 +65,6 @@ def lambda_handler(event, context):
 
     # else GET, display stats
     elif getpost == "get":
-
         try:
             table = db.Table("trd_statistics")
             results = []

diff --git a/examples/tz1boot1pK9h2BVGXdyvfQSv8kd1LQM6H889.yaml b/examples/tz1boot1pK9h2BVGXdyvfQSv8kd1LQM6H889.yaml
@@ -3,6 +3,7 @@ baking_address: tz1boot1pK9h2BVGXdyvfQSv8kd1LQM6H889
 payment_address: KT1KLQbYFtFZ5mAEnfEMZaWYuNtCsGuP5cLS
 rewards_type: actual
 service_fee: 4.5
+tzpro_api_key: XXXXXXXX
 founders_map:
   {'tz1boot1pK9h2BVGXdyvfQSv8kd1LQM6H889' : 0.3,
    'KT1KLQbYFtFZ5mAEnfEMZaWYuNtCsGuP5cLS' : 0.7}

diff --git a/readme.md b/readme.md
@@ -16,17 +16,15 @@ The documentation can be found [here](https://tezos-reward-distributor-organizat
 
 You can also ask for support on the TRD channel of the Tezos-Baking Slack group, if you are a member of this group you can view the channel [here](https://tezos-baking.slack.com/messages/CQ35AM8KE), if you are not a member you can [join the group](https://join.slack.com/t/tezos-baking/shared_invite/zt-yqxeszcy-rpvYmBtXr5oewh6M0DFkyQ) and find the trd channel from the channel list or simply [file an issue](https://github.com/tezos-reward-distributor-organization/tezos-reward-distributor/issues).
 
-TRD supports complex payments, pays in batches, and supports three backends for calculations: Tezos RPC, [TzStats API](https://tzstats.com/docs/api/) and [TzKT API](https://api.tzkt.io/). TRD is developed and tested extensively by the community.
+TRD supports complex payments, pays in batches, and supports three backends for calculations: Tezos RPC, [TZPRO API](https://docs.tzpro.io/) and [TzKT API](https://api.tzkt.io/). TRD is developed and tested extensively by the community.
 
 **Provider notes:**
 
-### TzStats
+### Blockwatch: TZPRO
 
-The [terms and conditions](https://tzstats.com/terms) of TzStats note that a license is needed for the commercial use of the API.
+The [terms and conditions](https://tzpro.io/terms) of TZPRO note that an account and API key are needed for the use of the API. Please review the [pricing](https://tzpro.io/#pricing) information. For further help contact hello@blockwatch.cc for more information.
 
-> If you wish to use the Data in a manner that is primarily intended for or directed towards commercial advantage or monetary compensation (such use, “Commercial Use”), Blockwatch requires that you enter into a separate commercial license agreement. Entering into a separate commercial license allows us to protect Blockwatch’s investment in the Data and to maintain the integrity of the Data.
->
-> Please contact us at license@blockwatch.cc for more information about Commercial Uses of our Data.
+In order to use your API key in the application copy and rename the .env.example to .env and add the API key for TZPRO.
 
 ### TzKT
 
@@ -106,4 +104,4 @@ TRD necessitates of an interface to get provided with income and delegator data
 The default provider is the TzKT API. However, it is possible to change the data provider to a local node with the flag `-P rpc`.
 In this case, the default node would be `127.0.0.1:8732`. In order to change the node URL for the provider, you can pass it in the form `node_url:port` using the flag `-A` (e.g. `-P rpc -A 127.0.0.1:8733`). Please note that the node should be an [archive node](https://tezos.gitlab.io/user/history_modes.html#setting-up-a-node-in-archive-mode), and that the port should be the RPC port specified while launching the node.
 
-It is also possible to use a public RPC node with flag `-P prpc`, which defaults to `https://rpc.tzstats.com`.
+It is also possible to use a public RPC node with flag `-P prpc`, which defaults to `https://mainnet.smartpy.io`.
diff --git a/requirements.txt b/requirements.txt
@@ -1,9 +1,9 @@
 # User requirements
-requests>=2.27.0
-base58>=1.0.2
-pyyaml>=5.3.1
+requests>=2.31.0
+base58>=2.1.1
+pyyaml>=6.0.1
 fysom>=2.1.6
-parse>=1.19.0
-transitions>=0.8.11
-tweepy>=4.6.0
-pip>=22.0.4
+parse>=1.19.1
+transitions>=0.9.0
+tweepy>=4.14.0
+pip>=23.3.1
diff --git a/requirements_developers.txt b/requirements_developers.txt
@@ -1,13 +1,13 @@
 # developer requirements
-pytest>=7.0.1
+pytest>=7.4.3
 pytest-watch>=4.2.0
-parameterized>=0.8.1
-Pygments>=2.11.2
-graphviz>=0.19.1
-click==8.0.4
-black==22.10.0
-flake8>=5.0.0
+parameterized>=0.9.0
+Pygments>=2.16.1
+graphviz>=0.20.1
+click==8.1.7
+black==23.3.0
+flake8>=5.0.4
 recommonmark==0.7.1
-sphinx_rtd_theme==1.0.0
-sphinxcontrib.spelling>=7.3.2
-pre-commit>=2.20.0
+sphinx_rtd_theme==1.3.0
+sphinxcontrib.spelling>=8.0.0
+pre-commit>=2.21.0
diff --git a/service_add.py b/service_add.py
@@ -2,7 +2,7 @@
 import sys
 import argparse
 from src.Constants import BASE_DIR, CONFIG_DIR
-from util.exit_program import exit_program, ExitCode
+from src.util.exit_program import exit_program, ExitCode
 
 
 def command_line_arguments():

diff --git a/src/Constants.py b/src/Constants.py
@@ -8,7 +8,7 @@
 LINER = "--------------------------------------------"
 
 # Disabled or enabled options by developers
-ALLOWED_REWARD_DATA_PROVIDER_CHOICES = ["tzkt", "tzstats"]
+ALLOWED_REWARD_DATA_PROVIDER_CHOICES = ["tzkt", "tzpro"]
 
 # Persistent data directories
 BASE_DIR = "~/pymnt"
@@ -55,10 +55,10 @@
     CURRENT_TESTNET: "https://testnet.smartpy.io",
 }
 
-# TzStats
-TZSTATS_PUBLIC_API_URL = {
-    "MAINNET": "https://api.tzstats.com",
-    CURRENT_TESTNET: "https://api.{}.tzstats.com".format(TESTNET_PREFIX.lower()),
+# TzPro
+TZPRO_API_URL = {
+    "MAINNET": "https://api.tzpro.io",
+    CURRENT_TESTNET: "https://api.{}.tzpro.io".format(TESTNET_PREFIX.lower()),
 }
 
 # TzKT
@@ -163,9 +163,24 @@ class PaymentStatus(Enum):
     INJECTED = 3
     AVOIDED = 4
 
+    def is_undefined(self):
+        return self.value == -1
+
     def is_fail(self):
         return self.value == 0
 
+    def is_paid(self):
+        return self.value == 1
+
+    def is_done(self):
+        return self.value == 2
+
+    def is_injected(self):
+        return self.value == 3
+
+    def is_avoided(self):
+        return self.value == 4
+
     def is_processed(self):
         return self.value > 0
 

diff --git a/src/Dexter/dexter_utils.py b/src/Dexter/dexter_utils.py
@@ -20,7 +20,6 @@ def get_dexter_balance_map(contract_id, snapshot_block, api_provider):
 def process_original_delegators_map(
     delegator_map, contract_id, snapshot_block, api_provider
 ):
-
     contract_balance = delegator_map[contract_id]["staking_balance"]
     dexter_liquidity_provider_map, totalLiquidity = get_dexter_balance_map(
         contract_id, snapshot_block, api_provider

diff --git a/src/api/provider_factory.py b/src/api/provider_factory.py
@@ -1,7 +1,7 @@
 from rpc.rpc_block_api import RpcBlockApiImpl
 from rpc.rpc_reward_api import RpcRewardApiImpl
-from tzstats.tzstats_block_api import TzStatsBlockApiImpl
-from tzstats.tzstats_reward_api import TzStatsRewardApiImpl
+from blockwatch.tzpro_block_api import TzProBlockApiImpl
+from blockwatch.tzpro_reward_api import TzProRewardApiImpl
 from tzkt.tzkt_block_api import TzKTBlockApiImpl
 from tzkt.tzkt_reward_api import TzKTRewardApiImpl
 from Constants import PUBLIC_NODE_URL
@@ -18,6 +18,7 @@ def newRewardApi(
         node_url,
         node_url_public="",
         api_base_url=None,
+        tzpro_api_key="",
     ):
         if self.provider == "rpc":
             if node_url.find("http") == -1:
@@ -27,22 +28,28 @@ def newRewardApi(
             if node_url_public == "":
                 node_url_public = PUBLIC_NODE_URL[network_config["NAME"]]
             return RpcRewardApiImpl(network_config, baking_address, node_url_public)
-        elif self.provider == "tzstats":
-            return TzStatsRewardApiImpl(network_config, baking_address)
+        elif self.provider == "tzpro":
+            return TzProRewardApiImpl(network_config, baking_address, tzpro_api_key)
         elif self.provider == "tzkt":
             return TzKTRewardApiImpl(
                 network_config, baking_address, base_url=api_base_url
             )
 
         raise Exception("No supported reward data provider : {}".format(self.provider))
 
-    def newBlockApi(self, network_config, node_url, api_base_url=None):
+    def newBlockApi(
+        self,
+        network_config,
+        node_url,
+        api_base_url=None,
+        tzpro_api_key="",
+    ):
         if self.provider == "rpc" or self.provider == "prpc":
             if node_url.find("http") == -1:
                 node_url = "http://" + node_url
             return RpcBlockApiImpl(network_config, node_url)
-        elif self.provider == "tzstats":
-            return TzStatsBlockApiImpl(network_config)
+        elif self.provider == "tzpro":
+            return TzProBlockApiImpl(network_config, tzpro_api_key)
         elif self.provider == "tzkt":
             return TzKTBlockApiImpl(network_config, base_url=api_base_url)
 

diff --git a/src/tzstats/__init__.py → src/blockwatch/__init__.py b/src/tzstats/__init__.py → src/blockwatch/__init__.py
diff --git a/src/blockwatch/tzpro_api_constants.py b/src/blockwatch/tzpro_api_constants.py
@@ -0,0 +1,37 @@
+# TODO: Check if there are changes to the indicies!
+# https://docs.tzpro.io/api/index/tables/income-table?tab=Response
+
+# Rights
+idx_income_baking_rights = 8
+idx_income_blocks_baked = 14
+idx_income_blocks_not_baked = 16
+idx_income_active_stake = 6
+
+# Income/Rewards Breakdown
+idx_income_expected_income = 21
+idx_income_total_income = 22
+idx_income_total_deposits = 23
+idx_income_baking_income = 24
+idx_income_endorsing_income = 25
+idx_income_accusation_income = 26
+idx_income_seed_income = 27
+idx_income_fees_income = 28
+idx_income_total_loss = 29
+idx_income_accusation_loss = 30
+idx_income_seed_loss = 31
+idx_income_endorsing_loss = 32
+idx_income_lost_accusation_fees = 33
+idx_income_lost_accusation_rewards = 34
+idx_income_lost_accusation_deposits = 35
+idx_income_lost_seed_fees = 36
+idx_income_lost_seed_rewards = 37
+
+# Cycle Snapshot
+idx_balance = 0
+idx_baker_delegated = 1
+idx_delegator_address = 2
+
+# Current balances
+idx_cb_delegator_id = 0
+idx_cb_current_balance = 1
+idx_cb_delegator_address = 2