Staging into Release branch (#1275)

* (un)Staking multiple avoid tx limit (#1244)

* add tx rate limit

* wait for tx limit if not done multi stake/unstake

* dont "decrypt" hotkey

* additional logging for prometheus (#1246)

* Dataset fix (#1249)

* fix

* added try except

* Grab delegates details from GitHub (#1245)

* add url to init

* add dataclass and util functions

* use in cli

* remove delegates json

---------

Co-authored-by: joeylegere <joeylegere@gmail.com>

* Add raw spec for local test and new bins (#1243)

* add spec and new bins

* fix config netuid

* use dot get

* check if config netuid is list

* add start to mockstatus

* add attr to mock neuron

* add info to mock from neurons

* change ordering of neuron dict to namespace

* remove test for wandb for axon

* use regex for looser match

* fix blacklist metagraph mock

* use real mock netuid

* use mock network and netuid in constructor

* fix patch

* patch delegate check

* use mock network and netuid

* remove check for wallet hotkey

* fix tests for subtensor init

* dont set netuid for overview test

* typo in docstring

* add mock status stop

* add low mock tx limit

* oops typo

* use dot get

* add wait for final and incl args

* use args during setup

* update bins and use 100ms blocktime

* pass block arg

* remove bittensor.logging and a old test

* use random port

* backward fix

* fix block time to 1s

* compile no symb on linux

* compile no symb mac

* remove useless init on var

* use dot get for new flags

* update test durations

* update test durations

* use dot get for config

* output error msg

* mock to_default

* remove to defaults in help

* reduce neruons, remove flaky test

* deactivate test

* mvoe key pair tests out of the subtensor interface

---------

Co-authored-by: Eugene <etesting007@gmail.com>

* Fix list_delegates on non-archive nodes (#1232)

* Change how pull of archival data is handled

* fix for list_delegates too

* .

* use empty dict

* fix spacing

* specify exception

* log out

* add space in log message

* use warning instead

* Blacklist fixes + depreciation of old signatures (#1240)

* fixes blacklist error message + remove

* remove checks for parse signature

* remove sign v1 tests

* fix for the syanpse checks

* fix tests and remove duplicate sign

* [BIT-636] Change u16 weight normalization to max-upscaling (#1241)

* Change u16 weight normalization to max-upscaling

Use full u16 bitwidth so that max_weight=U16_MAX, then rely on subtensor epoch to properly normalize weights in I32F32. This means that weights submission extrinsic to subtensor does not have to be pre-normalized.

* Skip zero sum in weight conversion

* Round u16 weights

* remove duplicate command #1228 (#1231)

* remove duplicate command #1228

* Extract create_parser for cli testing

* mark as private

* use in tests and test for duplicates

* fix test using mock prompt answers

* test_forward_priority_2nd_request_timeout fix (#1276)

fix

* Remove btcli query and btcli set_weights (#1144)

.

---------

Co-authored-by: Eugene-hu <85906264+Eugene-hu@users.noreply.github.com>
Co-authored-by: isabella618033 <49876827+isabella618033@users.noreply.github.com>
Co-authored-by: joeylegere <joeylegere@gmail.com>
Co-authored-by: Eugene <etesting007@gmail.com>
Co-authored-by: opentaco <93473497+opentaco@users.noreply.github.com>

bittensor/__init__.py

@@ -59,6 +59,9 @@ def turn_console_off():
 # Pip address for versioning
 __pipaddress__ = 'https://pypi.org/pypi/bittensor/json'
 
+# Raw github url for delegates registry file
+__delegates_details_url__: str = "https://raw.githubusercontent.com/opentensor/bittensor-delegates/main/public/delegates.json"
+
 # Substrate ss58_format
 __ss58_format__ = 42
 

bittensor/_axon/__init__.py

@@ -374,22 +374,6 @@ def __init__(
         self.blacklist = blacklist
         self.receiver_hotkey = receiver_hotkey
 
-    def parse_legacy_signature(
-        self, signature: str
-    ) -> Union[Tuple[int, str, str, str, int], None]:
-        r"""Attempts to parse a signature using the legacy format, using `bitxx` as a separator"""
-        parts = signature.split("bitxx")
-        if len(parts) < 4:
-            return None
-        try:
-            nonce = int(parts[0])
-            parts = parts[1:]
-        except ValueError:
-            return None
-        receptor_uuid, parts = parts[-1], parts[:-1]
-        signature, parts = parts[-1], parts[:-1]
-        sender_hotkey = "".join(parts)
-        return (nonce, sender_hotkey, signature, receptor_uuid, 1)
 
     def parse_signature_v2(
         self, signature: str
@@ -405,7 +389,7 @@ def parse_signature_v2(
         sender_hotkey = parts[1]
         signature = parts[2]
         receptor_uuid = parts[3]
-        return (nonce, sender_hotkey, signature, receptor_uuid, 2)
+        return (nonce, sender_hotkey, signature, receptor_uuid)
 
     def parse_signature(
         self, metadata: Dict[str, str]
@@ -418,10 +402,9 @@ def parse_signature(
         if int(version) < 370:
             raise Exception("Incorrect Version")
 
-        for parser in [self.parse_signature_v2, self.parse_legacy_signature]:
-            parts = parser(signature)
-            if parts is not None:
-                return parts
+        parts = self.parse_signature_v2(signature)
+        if parts is not None:
+            return parts
         raise Exception("Unknown signature format")
 
     def check_signature(
@@ -430,17 +413,12 @@ def check_signature(
         sender_hotkey: str,
         signature: str,
         receptor_uuid: str,
-        format: int,
     ):
         r"""verification of signature in metadata. Uses the pubkey and nonce"""
         keypair = Keypair(ss58_address=sender_hotkey)
         # Build the expected message which was used to build the signature.
-        if format == 2:
-            message = f"{nonce}.{sender_hotkey}.{self.receiver_hotkey}.{receptor_uuid}"
-        elif format == 1:
-            message = f"{nonce}{sender_hotkey}{receptor_uuid}"
-        else:
-            raise Exception("Invalid signature version")
+        message = f"{nonce}.{sender_hotkey}.{self.receiver_hotkey}.{receptor_uuid}"
+
         # Build the key which uniquely identifies the endpoint that has signed
         # the message.
         endpoint_key = f"{sender_hotkey}:{receptor_uuid}"
@@ -467,8 +445,10 @@ def black_list_checking(self, hotkey: str, method: str):
         if request_type is None:
             raise Exception("Unknown request type")
 
-        if self.blacklist(hotkey, request_type):
-            raise Exception("Request type is blacklisted")
+        failed, error_message =  self.blacklist(hotkey, request_type)
+        if failed:
+            raise Exception(str(error_message))
+
 
     def intercept_service(self, continuation, handler_call_details):
         r"""Authentication between bittensor nodes. Intercepts messages and checks them"""
@@ -481,12 +461,11 @@ def intercept_service(self, continuation, handler_call_details):
                 sender_hotkey,
                 signature,
                 receptor_uuid,
-                signature_format,
             ) = self.parse_signature(metadata)
 
             # signature checking
             self.check_signature(
-                nonce, sender_hotkey, signature, receptor_uuid, signature_format
+                nonce, sender_hotkey, signature, receptor_uuid
             )
 
             # blacklist checking

bittensor/_cli/__init__.py

@@ -59,11 +59,10 @@ def __new__(
             return naka_CLI(config=config)
         else:
             return cli_impl.CLI( config = config)
-
-    @staticmethod   
-    def config(args: List[str]) -> 'bittensor.config':
-        """ From the argument parser, add config to bittensor.executor and local config 
-            Return: bittensor.config object
+
+    @staticmethod
+    def __create_parser__() -> 'argparse.ArgumentParser':
+        """ Creates the argument parser for the bittensor cli.
         """
         parser = argparse.ArgumentParser(
             description=f"bittensor cli v{bittensor.__version__}",
@@ -74,7 +73,6 @@ def config(args: List[str]) -> 'bittensor.config':
         RunCommand.add_args( cmd_parsers )
         HelpCommand.add_args( cmd_parsers ) 
         ListCommand.add_args( cmd_parsers )
-        QueryCommand.add_args( cmd_parsers )
         StakeCommand.add_args( cmd_parsers )
         UpdateCommand.add_args( cmd_parsers )
         InspectCommand.add_args( cmd_parsers ) 
@@ -86,9 +84,7 @@ def config(args: List[str]) -> 'bittensor.config':
         NominateCommand.add_args( cmd_parsers )
         NewHotkeyCommand.add_args( cmd_parsers )
         MetagraphCommand.add_args( cmd_parsers )
-        SetWeightsCommand.add_args( cmd_parsers )
         NewColdkeyCommand.add_args( cmd_parsers )
-        NewHotkeyCommand.add_args( cmd_parsers )
         MyDelegatesCommand.add_args( cmd_parsers )
         ListSubnetsCommand.add_args( cmd_parsers )
         RegenHotkeyCommand.add_args( cmd_parsers )
@@ -99,6 +95,15 @@ def config(args: List[str]) -> 'bittensor.config':
         RegenColdkeypubCommand.add_args( cmd_parsers )
         RecycleRegisterCommand.add_args( cmd_parsers )
 
+        return parser
+
+    @staticmethod   
+    def config(args: List[str]) -> 'bittensor.config':
+        """ From the argument parser, add config to bittensor.executor and local config 
+            Return: bittensor.config object
+        """
+        parser = cli.__create_parser__()
+
         # If no arguments are passed, print help text.
         if len(args) == 0:
             parser.print_help()
@@ -136,14 +141,10 @@ def check_config (config: 'bittensor.Config'):
             MetagraphCommand.check_config( config )
         elif config.command == "weights":
             WeightsCommand.check_config( config )
-        elif config.command == "set_weights":
-            SetWeightsCommand.check_config( config )
         elif config.command == "list":
             ListCommand.check_config( config )
         elif config.command == "inspect":
             InspectCommand.check_config( config )
-        elif config.command == "query":
-            QueryCommand.check_config( config )
         elif config.command == "help":
             HelpCommand.check_config( config )
         elif config.command == "update":

bittensor/_cli/cli_impl.py

@@ -68,12 +68,8 @@ def run ( self ):
             MetagraphCommand.run( self )
         elif self.config.command == "weights":
             WeightsCommand.run( self )
-        elif self.config.command == "set_weights":
-            SetWeightsCommand.run( self )
         elif self.config.command == "inspect":
             InspectCommand.run( self )
-        elif self.config.command == "query":
-            QueryCommand.run( self )
         elif self.config.command == "help":
             HelpCommand.run( self )
         elif self.config.command == 'update':

bittensor/_cli/commands/__init__.py

@@ -9,6 +9,5 @@
 from .inspect import InspectCommand
 from .metagraph import MetagraphCommand
 from .list import ListCommand
-from .weights import SetWeightsCommand, WeightsCommand
-from .query import QueryCommand
+from .weights import WeightsCommand
 from .misc import HelpCommand, UpdateCommand, ListSubnetsCommand

bittensor/_cli/commands/delegates.py

@@ -26,10 +26,12 @@
 from rich.prompt import Confirm
 from rich.console import Text
 from tqdm import tqdm
+from substrateinterface.exceptions import SubstrateRequestException
+from .utils import get_delegates_details, DelegatesDetails
 
 import os
 import bittensor
-from typing import List
+from typing import List, Dict, Optional
 
 def _get_coldkey_wallets_for_path( path: str ) -> List['bittensor.wallet']:
     try:
@@ -43,22 +45,18 @@ def _get_coldkey_wallets_for_path( path: str ) -> List['bittensor.wallet']:
 console = bittensor.__console__
 
 # Uses rich console to pretty print a table of delegates.
-def show_delegates( delegates: List['bittensor.DelegateInfo'], prev_delegates: List['bittensor.DelegateInfo'], width: Optional[int] = None):
+def show_delegates( delegates: List['bittensor.DelegateInfo'], prev_delegates: Optional[List['bittensor.DelegateInfo']], width: Optional[int] = None):
     """ Pretty prints a table of delegates sorted by total stake.
     """
     delegates.sort(key=lambda delegate: delegate.total_stake, reverse=True)
     prev_delegates_dict = {}
-    for prev_delegate in prev_delegates:
-        prev_delegates_dict[prev_delegate.hotkey_ss58] = prev_delegate
-    try:
-        package_dir = os.path.dirname(bittensor.__file__)
-        root_dir = os.path.dirname(package_dir)
-        filename = os.path.join(root_dir, 'delegates.json')
-        if os.path.exists(filename):
-            registered_delegate_info = json.load( open(filename, 'r') )
-        else:
-            registered_delegate_info = {}
-    except:
+    if prev_delegates is not None:
+        for prev_delegate in prev_delegates:
+            prev_delegates_dict[prev_delegate.hotkey_ss58] = prev_delegate
+
+    registered_delegate_info: Optional[Dict[str, DelegatesDetails]] = get_delegates_details(url = bittensor.__delegates_details_url__)
+    if registered_delegate_info is None:
+        bittensor.__console__.print( ':warning:[yellow]Could not get delegate info from chain.[/yellow]')
         registered_delegate_info = {}
 
     table = Table(show_footer=True, width=width, pad_edge=False, box=None, expand=True)
@@ -85,9 +83,9 @@ def show_delegates( delegates: List['bittensor.DelegateInfo'], prev_delegates: L
             bittensor.Balance.from_rao(0) # default to 0 if no owner stake.
         )
         if delegate.hotkey_ss58 in registered_delegate_info:
-            delegate_name = registered_delegate_info[delegate.hotkey_ss58]['name']
-            delegate_url = registered_delegate_info[delegate.hotkey_ss58]['url']
-            delegate_description =  registered_delegate_info[delegate.hotkey_ss58]['description']
+            delegate_name = registered_delegate_info[delegate.hotkey_ss58].name
+            delegate_url = registered_delegate_info[delegate.hotkey_ss58].url
+            delegate_description =  registered_delegate_info[delegate.hotkey_ss58].description
         else:
             delegate_name = ''
             delegate_url = ''
@@ -106,7 +104,7 @@ def show_delegates( delegates: List['bittensor.DelegateInfo'], prev_delegates: L
                 else:
                     rate_change_in_stake_str = "[grey0]0%[/grey0]"
         else:
-            rate_change_in_stake_str = "[grey0]0%[/grey0]"
+            rate_change_in_stake_str = "[grey0]NA[/grey0]"
 
         table.add_row(
             str(i),
@@ -190,10 +188,16 @@ def check_config( config: 'bittensor.Config' ):
             with bittensor.__console__.status(":satellite: Loading delegates..."):
                 subtensor = bittensor.subtensor( config = config )
                 delegates: List[bittensor.DelegateInfo] = subtensor.get_delegates()
-                prev_delegates = subtensor.get_delegates(max(0, subtensor.block - 1200))
+                try:
+                    prev_delegates = subtensor.get_delegates(max(0, subtensor.block - 1200))
+                except SubstrateRequestException:
+                    prev_delegates = None
+
+            if prev_delegates is None:
+                bittensor.__console__.print(":warning: [yellow]Could not fetch delegates history[/yellow]")
 
             if len(delegates) == 0:
-                console.print(":cross_mark:[red]There are no delegates on {}[/red]".format(subtensor.network))
+                console.print(":cross_mark: [red]There are no delegates on {}[/red]".format(subtensor.network))
                 sys.exit(1)
 
             delegates.sort(key=lambda delegate: delegate.total_stake, reverse=True)
@@ -213,7 +217,7 @@ def check_config( config: 'bittensor.Config' ):
                 try:
                     config.amount = float(amount)
                 except ValueError:
-                    console.print(":cross_mark:[red]Invalid Tao amount[/red] [bold white]{}[/bold white]".format(amount))
+                    console.print(":cross_mark: [red]Invalid Tao amount[/red] [bold white]{}[/bold white]".format(amount))
                     sys.exit()
             else:
                 config.stake_all = True
@@ -289,10 +293,16 @@ def check_config( config: 'bittensor.Config' ):
             with bittensor.__console__.status(":satellite: Loading delegates..."):
                 subtensor = bittensor.subtensor( config = config )
                 delegates: List[bittensor.DelegateInfo] = subtensor.get_delegates()
-                prev_delegates = subtensor.get_delegates(max(0, subtensor.block - 1200))
+                try:
+                    prev_delegates = subtensor.get_delegates(max(0, subtensor.block - 1200))
+                except SubstrateRequestException:
+                    prev_delegates = None
+
+            if prev_delegates is None:
+                bittensor.__console__.print(":warning: [yellow]Could not fetch delegates history[/yellow]")
 
             if len(delegates) == 0:
-                console.print(":cross_mark:[red]There are no delegates on {}[/red]".format(subtensor.network))
+                console.print(":cross_mark: [red]There are no delegates on {}[/red]".format(subtensor.network))
                 sys.exit(1)
 
             delegates.sort(key=lambda delegate: delegate.total_stake, reverse=True)
@@ -308,7 +318,7 @@ def check_config( config: 'bittensor.Config' ):
                 try:
                     config.amount = float(amount)
                 except ValueError:
-                    console.print(":cross_mark:[red]Invalid Tao amount[/red] [bold white]{}[/bold white]".format(amount))
+                    console.print(":cross_mark: [red]Invalid Tao amount[/red] [bold white]{}[/bold white]".format(amount))
                     sys.exit()
             else:
                 config.unstake_all = True
@@ -323,7 +333,14 @@ def run( cli ):
         subtensor = bittensor.subtensor( config = cli.config )
         with bittensor.__console__.status(":satellite: Loading delegates..."):
             delegates: bittensor.DelegateInfo = subtensor.get_delegates()
-            prev_delegates = subtensor.get_delegates(max(0, subtensor.block - 1200))
+            try:
+                prev_delegates = subtensor.get_delegates(max(0, subtensor.block - 1200))
+            except SubstrateRequestException:
+                prev_delegates = None
+
+        if prev_delegates is None:
+            bittensor.__console__.print(":warning: [yellow]Could not fetch delegates history[/yellow]")
+
         show_delegates( delegates, prev_delegates = prev_delegates, width = cli.config.get('width', None) )
 
     @staticmethod
@@ -409,7 +426,7 @@ class MyDelegatesCommand:
     def run( cli ):
         '''Delegates stake to a chain delegate.'''
         config = cli.config.copy()
-        if config.all == True:
+        if config.get('all', d=None) == True:
             wallets = _get_coldkey_wallets_for_path( config.wallet.path )
         else:
             wallets = [bittensor.wallet( config = config )]
@@ -441,15 +458,9 @@ def run( cli ):
 
             delegates.sort(key=lambda delegate: delegate[0].total_stake, reverse=True)
 
-            try:
-                package_dir = os.path.dirname(bittensor.__file__)
-                root_dir = os.path.dirname(package_dir)
-                filename = os.path.join(root_dir, 'delegates.json')
-                if os.path.exists(filename):
-                    registered_delegate_info = json.load( open(filename, 'r') )
-                else:
-                    registered_delegate_info = {}
-            except:
+            registered_delegate_info: Optional[DelegatesDetails] = get_delegates_details(url = bittensor.__delegates_details_url__)
+            if registered_delegate_info is None:
+                bittensor.__console__.print( ':warning:[yellow]Could not get delegate info from chain.[/yellow]')
                 registered_delegate_info = {}
 
             for i, delegate in enumerate( delegates ):
@@ -518,7 +529,7 @@ def add_args( parser: argparse.ArgumentParser ):
 
     @staticmethod   
     def check_config( config: 'bittensor.Config' ):
-        if not config.all and config.wallet.get('name') == bittensor.defaults.wallet.name and not config.no_prompt:
+        if not config.get( 'all', d=None ) and config.wallet.get('name') == bittensor.defaults.wallet.name and not config.no_prompt:
             wallet_name = Prompt.ask("Enter wallet name", default = bittensor.defaults.wallet.name)
             config.wallet.name = str(wallet_name)