From add832501162b5ac70aa30cfa90a89ad8ed8ad53 Mon Sep 17 00:00:00 2001 From: philanthrope Date: Tue, 31 Oct 2023 15:07:41 -0400 Subject: [PATCH] Master into staging (#1570) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Release/6.2.0 (#1567) * (un)Staking multiple avoid tx limit (#1244) * add tx rate limit * wait for tx limit if not done multi stake/unstake * dont "decrypt" hotkey * additional logging for prometheus * additional logging for prometheus (#1246) * Dataset fix (#1249) * fix * added try except * Grab delegates details from GitHub (#1245) * add url to init * add dataclass and util functions * use in cli * remove delegates json --------- * Add raw spec for local test and new bins (#1243) * add spec and new bins * fix config netuid * use dot get * check if config netuid is list * add start to mockstatus * add attr to mock neuron * add info to mock from neurons * change ordering of neuron dict to namespace * remove test for wandb for axon * use regex for looser match * fix blacklist metagraph mock * use real mock netuid * use mock network and netuid in constructor * fix patch * patch delegate check * use mock network and netuid * remove check for wallet hotkey * fix tests for subtensor init * dont set netuid for overview test * typo in docstring * add mock status stop * add low mock tx limit * oops typo * use dot get * add wait for final and incl args * use args during setup * update bins and use 100ms blocktime * pass block arg * remove bittensor.logging and a old test * use random port * backward fix * fix block time to 1s * compile no symb on linux * compile no symb mac * remove useless init on var * use dot get for new flags * update test durations * update test durations * use dot get for config * output error msg * mock to_default * remove to defaults in help * reduce neruons, remove flaky test * deactivate test * mvoe key pair tests out of the subtensor interface --------- * Fix list_delegates on non-archive nodes (#1232) * Change how pull of archival data is handled * fix for list_delegates too * . * use empty dict * fix spacing * specify exception * log out * add space in log message * use warning instead * Blacklist fixes + depreciation of old signatures (#1240) * fixes blacklist error message + remove * remove checks for parse signature * remove sign v1 tests * fix for the syanpse checks * fix tests and remove duplicate sign * Improve development workflow documentation * [BIT-636] Change u16 weight normalization to max-upscaling (#1241) * Change u16 weight normalization to max-upscaling Use full u16 bitwidth so that max_weight=U16_MAX, then rely on subtensor epoch to properly normalize weights in I32F32. This means that weights submission extrinsic to subtensor does not have to be pre-normalized. * Skip zero sum in weight conversion * Round u16 weights * remove duplicate command #1228 (#1231) * remove duplicate command #1228 * Extract create_parser for cli testing * mark as private * use in tests and test for duplicates * fix test using mock prompt answers * test_forward_priority_2nd_request_timeout fix (#1276) fix * Remove btcli query and btcli set_weights (#1144) . * Merge releases 4.0.0 and 4.0.1 back to staging (#1306) * bump version * Fix permissions for release github script (#1224) * should be 4.1.0 * Revert "should be 4.1.0" This reverts commit 3db08ea24f4fc4775bd46858e6c77cfa165d85ed. * Staging into Release branch (#1275) * (un)Staking multiple avoid tx limit (#1244) * add tx rate limit * wait for tx limit if not done multi stake/unstake * dont "decrypt" hotkey * additional logging for prometheus (#1246) * Dataset fix (#1249) * fix * added try except * Grab delegates details from GitHub (#1245) * add url to init * add dataclass and util functions * use in cli * remove delegates json --------- * Add raw spec for local test and new bins (#1243) * add spec and new bins * fix config netuid * use dot get * check if config netuid is list * add start to mockstatus * add attr to mock neuron * add info to mock from neurons * change ordering of neuron dict to namespace * remove test for wandb for axon * use regex for looser match * fix blacklist metagraph mock * use real mock netuid * use mock network and netuid in constructor * fix patch * patch delegate check * use mock network and netuid * remove check for wallet hotkey * fix tests for subtensor init * dont set netuid for overview test * typo in docstring * add mock status stop * add low mock tx limit * oops typo * use dot get * add wait for final and incl args * use args during setup * update bins and use 100ms blocktime * pass block arg * remove bittensor.logging and a old test * use random port * backward fix * fix block time to 1s * compile no symb on linux * compile no symb mac * remove useless init on var * use dot get for new flags * update test durations * update test durations * use dot get for config * output error msg * mock to_default * remove to defaults in help * reduce neruons, remove flaky test * deactivate test * mvoe key pair tests out of the subtensor interface --------- * Fix list_delegates on non-archive nodes (#1232) * Change how pull of archival data is handled * fix for list_delegates too * . * use empty dict * fix spacing * specify exception * log out * add space in log message * use warning instead * Blacklist fixes + depreciation of old signatures (#1240) * fixes blacklist error message + remove * remove checks for parse signature * remove sign v1 tests * fix for the syanpse checks * fix tests and remove duplicate sign * [BIT-636] Change u16 weight normalization to max-upscaling (#1241) * Change u16 weight normalization to max-upscaling Use full u16 bitwidth so that max_weight=U16_MAX, then rely on subtensor epoch to properly normalize weights in I32F32. This means that weights submission extrinsic to subtensor does not have to be pre-normalized. * Skip zero sum in weight conversion * Round u16 weights * remove duplicate command #1228 (#1231) * remove duplicate command #1228 * Extract create_parser for cli testing * mark as private * use in tests and test for duplicates * fix test using mock prompt answers * test_forward_priority_2nd_request_timeout fix (#1276) fix * Remove btcli query and btcli set_weights (#1144) . --------- * Remove codecov (#1282) * Use alt new preseal (#1269) * use new preseal for reg * bump cubit req * fix arg order issue * cubit req back * use alt impl * fix typehint * use 512 * modify tests for new format * refactor functions to use helpers and remove useless * refactor functions * add test for CPU solver * modify tests for privitized module and methods * private register cuda * move formatting funcs * use powsolution * privitize most methods * fix test * fix perms * remove test script * remove debug * fix call * fix seal * fix combined hash * move to method * fix test using real example * update mock bins * use new builder * fix block update tests * fix some patching in tests * mock live display for some tests * fix chain mock * update linux bin * add mock network flag * set max diff at 0 for mock netuid 1 * set min diff too * add try catch for setup * add some logging during tests * don't submit on cli register * update test durations * fix test to use mock keypair * return mock wallet * should use subtensor instance during rereg * update node subtensor bins * use fixtures and multiple subtensor instances * changelog update * skip CLI tests (#1284) * skip tests * dont test mock functions * update test durations * [Release] v4.0.0 (#1271) * bump version * Fix permissions for release github script (#1224) * should be 4.1.0 * Revert "should be 4.1.0" This reverts commit 3db08ea24f4fc4775bd46858e6c77cfa165d85ed. * Staging into Release branch (#1275) * (un)Staking multiple avoid tx limit (#1244) * add tx rate limit * wait for tx limit if not done multi stake/unstake * dont "decrypt" hotkey * additional logging for prometheus (#1246) * Dataset fix (#1249) * fix * added try except * Grab delegates details from GitHub (#1245) * add url to init * add dataclass and util functions * use in cli * remove delegates json --------- * Add raw spec for local test and new bins (#1243) * add spec and new bins * fix config netuid * use dot get * check if config netuid is list * add start to mockstatus * add attr to mock neuron * add info to mock from neurons * change ordering of neuron dict to namespace * remove test for wandb for axon * use regex for looser match * fix blacklist metagraph mock * use real mock netuid * use mock network and netuid in constructor * fix patch * patch delegate check * use mock network and netuid * remove check for wallet hotkey * fix tests for subtensor init * dont set netuid for overview test * typo in docstring * add mock status stop * add low mock tx limit * oops typo * use dot get * add wait for final and incl args * use args during setup * update bins and use 100ms blocktime * pass block arg * remove bittensor.logging and a old test * use random port * backward fix * fix block time to 1s * compile no symb on linux * compile no symb mac * remove useless init on var * use dot get for new flags * update test durations * update test durations * use dot get for config * output error msg * mock to_default * remove to defaults in help * reduce neruons, remove flaky test * deactivate test * mvoe key pair tests out of the subtensor interface --------- * Fix list_delegates on non-archive nodes (#1232) * Change how pull of archival data is handled * fix for list_delegates too * . * use empty dict * fix spacing * specify exception * log out * add space in log message * use warning instead * Blacklist fixes + depreciation of old signatures (#1240) * fixes blacklist error message + remove * remove checks for parse signature * remove sign v1 tests * fix for the syanpse checks * fix tests and remove duplicate sign * [BIT-636] Change u16 weight normalization to max-upscaling (#1241) * Change u16 weight normalization to max-upscaling Use full u16 bitwidth so that max_weight=U16_MAX, then rely on subtensor epoch to properly normalize weights in I32F32. This means that weights submission extrinsic to subtensor does not have to be pre-normalized. * Skip zero sum in weight conversion * Round u16 weights * remove duplicate command #1228 (#1231) * remove duplicate command #1228 * Extract create_parser for cli testing * mark as private * use in tests and test for duplicates * fix test using mock prompt answers * test_forward_priority_2nd_request_timeout fix (#1276) fix * Remove btcli query and btcli set_weights (#1144) . --------- * Remove codecov (#1282) * Use alt new preseal (#1269) * use new preseal for reg * bump cubit req * fix arg order issue * cubit req back * use alt impl * fix typehint * use 512 * modify tests for new format * refactor functions to use helpers and remove useless * refactor functions * add test for CPU solver * modify tests for privitized module and methods * private register cuda * move formatting funcs * use powsolution * privitize most methods * fix test * fix perms * remove test script * remove debug * fix call * fix seal * fix combined hash * move to method * fix test using real example * update mock bins * use new builder * fix block update tests * fix some patching in tests * mock live display for some tests * fix chain mock * update linux bin * add mock network flag * set max diff at 0 for mock netuid 1 * set min diff too * add try catch for setup * add some logging during tests * don't submit on cli register * update test durations * fix test to use mock keypair * return mock wallet * should use subtensor instance during rereg * update node subtensor bins * use fixtures and multiple subtensor instances * changelog update * skip CLI tests (#1284) * skip tests * dont test mock functions * update test durations --------- * fix my delegates * fix perms on changelog script * update version * fix changelog script * Catch bad endpoint protocol (#1296) * catch protocol not good * add protocol 4 * catch assertion and return bool * catch assertion errors * changelog --------- * Update DEVELOPMENT_WORKFLOW.md * final fixes * staging updates and fixes (#1540) * fix cli test * fix double-counted hotkeys per subnet and non-iterable stake obj (#1539) * fix double-counted hotkeys per subnet and non-iterable stake obj * run black * Add root get_weights command to btcli (#1536) * Add root get_weights command to btcli * Use a percentage for viewing weights instead of float values * run black, fix cli test --------- * Fix typo (#1543) * remove duplicated debug message in dendrite (#1544) * Cli fix (#1541) don't break on mismatched coldkey from local wallet <> chain * update faucet helpstr (#1542) * Added mechanism to sum all delegated tao (#1547) * Dict hash fix (#1548) * use dict() when hasing body objects to not convert arbitrary objects to str * recreate synapse in axon dependency to avoid duplicating code * black * Merge master (#1552) Release/6.1.0 (#1550) * Fix typo (#1543) * remove duplicated debug message in dendrite (#1544) * Cli fix (#1541) don't break on mismatched coldkey from local wallet <> chain * update faucet helpstr (#1542) * Added mechanism to sum all delegated tao (#1547) * Dict hash fix (#1548) * use dict() when hasing body objects to not convert arbitrary objects to str * recreate synapse in axon dependency to avoid duplicating code * black * update version * update changelog --------- * Streaming fix (#1551) * yield chunks immediately in process_streaming_responss so clients can access * break streaming into separate call funcs * update docstrings, types * black * duplicate debug msg * add warning for mismatched streaming arg + subclass * Fix typos (#1553) * Release/6.1.0 (#1550) * Fix typo (#1543) * remove duplicated debug message in dendrite (#1544) * Cli fix (#1541) don't break on mismatched coldkey from local wallet <> chain * update faucet helpstr (#1542) * Added mechanism to sum all delegated tao (#1547) * Dict hash fix (#1548) * use dict() when hasing body objects to not convert arbitrary objects to str * recreate synapse in axon dependency to avoid duplicating code * black * update version * update changelog --------- * fix typos --------- * Normalize weights in r get weights table (#1556) * Normalize weights in r get weights table * use numpy and make table look nicer * apply black * add max 1 to prevent div0 --------- * Dendrite & Synapse updates and fixes (#1555) * remove tensor header objects to not overflow headers * update tests to reflect removal of tensor headers * ensure consistent method for calling in synapse methods * fix dendrite UnClosesedSession error * fix docstring and add tests for close/aclose * rm extra delete * run black * add default synapse dict() consistency test * call del on session after close_session(), fix tests * update dendrite dummy clsname * add dendrite.query finally block * fix test * rm root flag in metagraph (#1558) * rm root flag in metagraph * run black * typo * Max Faucet Runs == 3 (#1560) add exceptions * replace unknown wallet params (chain mismatch) with key values (#1559) * replace unknown wallet params (chain mismatch) with key values * run black * rm debug prints * Remove PoW registration cli and associated extrinsic (#1557) * Remove PoW registration cli and associated extrinsic * run black * no mo pow, no mo pow tests * remove now deprecated PoW reregister routine * remove deprecated tests * more test fixes * remove _do_pow call * return PoW but still kill reregister (unused) * run black * return test to networks choices in btcli, fix chain_endpoint selection * fix pow args * Add btcli wallet balance (#1564) * begin adding balance command * skip validator that hasn't set weights yet on root * finish balances command * formatter * issue warning for nonexistent coldkeypub.txt rather than break * Dendrite fixes (#1561) * make sure a session exists before trying to close it * don't double iterate over async generator, simply return it * black * less DRY violations * fix typehints * add versioning * update changelog * remove unused registration utils * fix typos --------- Co-authored-by: Cameron Fairchild Co-authored-by: Eugene Co-authored-by: Eugene-hu <85906264+Eugene-hu@users.noreply.github.com> Co-authored-by: isabella618033 <49876827+isabella618033@users.noreply.github.com> Co-authored-by: joeylegere Co-authored-by: Cameron Fairchild Co-authored-by: “quac88” <“mac2@thrasher.com”> Co-authored-by: opentaco <93473497+opentaco@users.noreply.github.com> Co-authored-by: Eduardo García Co-authored-by: Ayden Brewer Co-authored-by: Steffen Cruz Co-authored-by: Ala Shaabana Co-authored-by: Ala Shaabana Co-authored-by: omahs <73983677+omahs@users.noreply.github.com> Co-authored-by: Cameron Fairchild Co-authored-by: Cameron Fairchild --- CHANGELOG.md | 43 ++ VERSION | 2 +- bittensor/__init__.py | 2 +- bittensor/dendrite.py | 2 +- bittensor/metagraph.py | 2 +- bittensor/utils/registratrion_old.py | 1030 -------------------------- 6 files changed, 47 insertions(+), 1034 deletions(-) delete mode 100644 bittensor/utils/registratrion_old.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 83254fbd3d..0e606fd03c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,48 @@ # Changelog +## 6.2.0 / 2023-10-30 + +## What's Changed +* (un)Staking multiple avoid tx limit by @camfairchild in https://github.com/opentensor/bittensor/pull/1244 +* additional logging for prometheus by @Eugene-hu in https://github.com/opentensor/bittensor/pull/1246 +* Dataset fix by @isabella618033 in https://github.com/opentensor/bittensor/pull/1249 +* Grab delegates details from GitHub by @camfairchild in https://github.com/opentensor/bittensor/pull/1245 +* Add raw spec for local test and new bins by @camfairchild in https://github.com/opentensor/bittensor/pull/1243 +* Fix list_delegates on non-archive nodes by @camfairchild in https://github.com/opentensor/bittensor/pull/1232 +* Blacklist fixes + depreciation of old signatures by @Eugene-hu in https://github.com/opentensor/bittensor/pull/1240 +* [BIT-636] Change u16 weight normalization to max-upscaling by @opentaco in https://github.com/opentensor/bittensor/pull/1241 +* remove duplicate command #1228 by @camfairchild in https://github.com/opentensor/bittensor/pull/1231 +* test_forward_priority_2nd_request_timeout fix by @isabella618033 in https://github.com/opentensor/bittensor/pull/1276 +* Remove btcli query and btcli set_weights by @camfairchild in https://github.com/opentensor/bittensor/pull/1144 +* Merge releases 4.0.0 and 4.0.1 back to staging by @camfairchild in https://github.com/opentensor/bittensor/pull/1306 +* Improve development workflow documentation by @quac88 in https://github.com/opentensor/bittensor/pull/1262 +* staging updates and fixes by @ifrit98 in https://github.com/opentensor/bittensor/pull/1540 +* Add root get_weights command to btcli by @Rubberbandits in https://github.com/opentensor/bittensor/pull/1536 +* Fix typo by @steffencruz in https://github.com/opentensor/bittensor/pull/1543 +* remove duplicated debug message in dendrite by @ifrit98 in https://github.com/opentensor/bittensor/pull/1544 +* Cli fix by @ifrit98 in https://github.com/opentensor/bittensor/pull/1541 +* update faucet helpstr by @ifrit98 in https://github.com/opentensor/bittensor/pull/1542 +* Added mechanism to sum all delegated tao by @shibshib in https://github.com/opentensor/bittensor/pull/1547 +* Dict hash fix by @ifrit98 in https://github.com/opentensor/bittensor/pull/1548 +* Release/6.1.0 by @ifrit98 in https://github.com/opentensor/bittensor/pull/1550 +* Merge master by @ifrit98 in https://github.com/opentensor/bittensor/pull/1552 +* Streaming fix by @ifrit98 in https://github.com/opentensor/bittensor/pull/1551 +* Fix typos by @omahs in https://github.com/opentensor/bittensor/pull/1553 +* Normalize weights in r get weights table by @camfairchild in https://github.com/opentensor/bittensor/pull/1556 +* Dendrite & Synapse updates and fixes by @ifrit98 in https://github.com/opentensor/bittensor/pull/1555 +* rm root flag in metagraph by @ifrit98 in https://github.com/opentensor/bittensor/pull/1558 +* Max Faucet Runs == 3 by @ifrit98 in https://github.com/opentensor/bittensor/pull/1560 +* replace unknown wallet params (chain mismatch) with key values by @ifrit98 in https://github.com/opentensor/bittensor/pull/1559 +* Remove PoW registration cli and associated extrinsic by @ifrit98 in https://github.com/opentensor/bittensor/pull/1557 +* Add btcli wallet balance by @ifrit98 in https://github.com/opentensor/bittensor/pull/1564 +* Dendrite fixes by @ifrit98 in https://github.com/opentensor/bittensor/pull/1561 + +## New Contributors +* @omahs made their first contribution in https://github.com/opentensor/bittensor/pull/1553 + +**Full Changelog**: https://github.com/opentensor/bittensor/compare/v6.0.1...v6.2.0 + + ## 6.1.0 / 2023-10-17 ## What's Changed diff --git a/VERSION b/VERSION index 358e78e607..4ac4fded49 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -6.1.0 \ No newline at end of file +6.2.0 \ No newline at end of file diff --git a/bittensor/__init__.py b/bittensor/__init__.py index 93f01c9d84..3215ce9155 100644 --- a/bittensor/__init__.py +++ b/bittensor/__init__.py @@ -27,7 +27,7 @@ nest_asyncio.apply() # Bittensor code and protocol version. -__version__ = "6.1.0" +__version__ = "6.2.0" version_split = __version__.split(".") __version_as_int__ = ( (100 * int(version_split[0])) diff --git a/bittensor/dendrite.py b/bittensor/dendrite.py index 6d9f546980..a99b28e300 100644 --- a/bittensor/dendrite.py +++ b/bittensor/dendrite.py @@ -256,7 +256,7 @@ async def forward( the requests, and then sends them off. After getting the responses, it processes and collates them into a unified format. - When querying an Axon that sends back data in chunks using the Dednrite, this function + When querying an Axon that sends back data in chunks using the Dendrite, this function returns an AsyncGenerator that yields each chunk as it is received. The generator can be iterated over to process each chunk individually. diff --git a/bittensor/metagraph.py b/bittensor/metagraph.py index c128dae983..789fb3b7a8 100644 --- a/bittensor/metagraph.py +++ b/bittensor/metagraph.py @@ -481,7 +481,7 @@ def _set_weights_and_bonds(self, subtensor: bittensor.subtensor = None): None. """ # TODO: Check and test the computation of weights and bonds - if self.netuid == 0: # Is this the root network? + if self.netuid == 0: self.weights = self._process_root_weights( [neuron.weights for neuron in self.neurons], "weights", subtensor ) diff --git a/bittensor/utils/registratrion_old.py b/bittensor/utils/registratrion_old.py deleted file mode 100644 index 453772d649..0000000000 --- a/bittensor/utils/registratrion_old.py +++ /dev/null @@ -1,1030 +0,0 @@ -import binascii -import hashlib -import math -import multiprocessing -import os -import random -import time -from dataclasses import dataclass -from datetime import timedelta -from queue import Empty, Full -from typing import Any, Callable, Dict, List, Optional, Tuple, Union - -import backoff -import bittensor -import torch -from Crypto.Hash import keccak -from rich import console as rich_console -from rich import status as rich_status - -from ._register_cuda import solve_cuda - - -class CUDAException(Exception): - """An exception raised when an error occurs in the CUDA environment.""" - - pass - - -def hex_bytes_to_u8_list(hex_bytes: bytes): - hex_chunks = [int(hex_bytes[i : i + 2], 16) for i in range(0, len(hex_bytes), 2)] - return hex_chunks - - -def u8_list_to_hex(values: list): - total = 0 - for val in reversed(values): - total = (total << 8) + val - return total - - -def create_seal_hash(block_hash: bytes, nonce: int) -> bytes: - block_bytes = block_hash.encode("utf-8")[2:] - nonce_bytes = binascii.hexlify(nonce.to_bytes(8, "little")) - pre_seal = nonce_bytes + block_bytes - seal_sh256 = hashlib.sha256(bytearray(hex_bytes_to_u8_list(pre_seal))).digest() - kec = keccak.new(digest_bits=256) - seal = kec.update(seal_sh256).digest() - return seal - - -def seal_meets_difficulty(seal: bytes, difficulty: int): - seal_number = int.from_bytes(seal, "big") - product = seal_number * difficulty - limit = int(math.pow(2, 256)) - 1 - if product > limit: - return False - else: - return True - - -def solve_for_difficulty(block_hash, difficulty): - meets = False - nonce = -1 - while not meets: - nonce += 1 - seal = create_seal_hash(block_hash, nonce) - meets = seal_meets_difficulty(seal, difficulty) - if nonce > 1: - break - return nonce, seal - - -def get_human_readable(num, suffix="H"): - for unit in ["", "K", "M", "G", "T", "P", "E", "Z"]: - if abs(num) < 1000.0: - return f"{num:3.1f}{unit}{suffix}" - num /= 1000.0 - return f"{num:.1f}Y{suffix}" - - -def millify(n: int): - millnames = ["", " K", " M", " B", " T"] - n = float(n) - millidx = max( - 0, - min( - len(millnames) - 1, int(math.floor(0 if n == 0 else math.log10(abs(n)) / 3)) - ), - ) - - return "{:.2f}{}".format(n / 10 ** (3 * millidx), millnames[millidx]) - - -def POWNotStale(subtensor: "bittensor.subtensor", pow_result: Dict) -> bool: - """Returns True if the POW is not stale. - This means the block the POW is solved for is within 3 blocks of the current block. - """ - return pow_result["block_number"] >= subtensor.get_current_block() - 3 - - -@dataclass -class POWSolution: - """A solution to the registration PoW problem.""" - - nonce: int - block_number: int - difficulty: int - seal: bytes - - -class SolverBase(multiprocessing.Process): - """ - A process that solves the registration PoW problem. - Args: - proc_num: int - The number of the process being created. - num_proc: int - The total number of processes running. - update_interval: int - The number of nonces to try to solve before checking for a new block. - finished_queue: multiprocessing.Queue - The queue to put the process number when a process finishes each update_interval. - Used for calculating the average time per update_interval across all processes. - solution_queue: multiprocessing.Queue - The queue to put the solution the process has found during the pow solve. - newBlockEvent: multiprocessing.Event - The event to set by the main process when a new block is finalized in the network. - The solver process will check for the event after each update_interval. - The solver process will get the new block hash and difficulty and start solving for a new nonce. - stopEvent: multiprocessing.Event - The event to set by the main process when all the solver processes should stop. - The solver process will check for the event after each update_interval. - The solver process will stop when the event is set. - Used to stop the solver processes when a solution is found. - curr_block: multiprocessing.Array - The array containing this process's current block hash. - The main process will set the array to the new block hash when a new block is finalized in the network. - The solver process will get the new block hash from this array when newBlockEvent is set. - curr_block_num: multiprocessing.Value - The value containing this process's current block number. - The main process will set the value to the new block number when a new block is finalized in the network. - The solver process will get the new block number from this value when newBlockEvent is set. - curr_diff: multiprocessing.Array - The array containing this process's current difficulty. - The main process will set the array to the new difficulty when a new block is finalized in the network. - The solver process will get the new difficulty from this array when newBlockEvent is set. - check_block: multiprocessing.Lock - The lock to prevent this process from getting the new block data while the main process is updating the data. - limit: int - The limit of the pow solve for a valid solution. - """ - - proc_num: int - num_proc: int - update_interval: int - finished_queue: multiprocessing.Queue - solution_queue: multiprocessing.Queue - newBlockEvent: multiprocessing.Event - stopEvent: multiprocessing.Event - curr_block: multiprocessing.Array - curr_block_num: multiprocessing.Value - curr_diff: multiprocessing.Array - check_block: multiprocessing.Lock - limit: int - - def __init__( - self, - proc_num, - num_proc, - update_interval, - finished_queue, - solution_queue, - stopEvent, - curr_block, - curr_block_num, - curr_diff, - check_block, - limit, - ): - multiprocessing.Process.__init__(self, daemon=True) - self.proc_num = proc_num - self.num_proc = num_proc - self.update_interval = update_interval - self.finished_queue = finished_queue - self.solution_queue = solution_queue - self.newBlockEvent = multiprocessing.Event() - self.newBlockEvent.clear() - self.curr_block = curr_block - self.curr_block_num = curr_block_num - self.curr_diff = curr_diff - self.check_block = check_block - self.stopEvent = stopEvent - self.limit = limit - - def run(self): - raise NotImplementedError("SolverBase is an abstract class") - - -class Solver(SolverBase): - def run(self): - block_number: int - block_bytes: bytes - block_difficulty: int - nonce_limit = int(math.pow(2, 64)) - 1 - - # Start at random nonce - nonce_start = random.randint(0, nonce_limit) - nonce_end = nonce_start + self.update_interval - while not self.stopEvent.is_set(): - if self.newBlockEvent.is_set(): - with self.check_block: - block_number = self.curr_block_num.value - block_bytes = bytes(self.curr_block) - block_difficulty = registration_diff_unpack(self.curr_diff) - - self.newBlockEvent.clear() - - # Do a block of nonces - solution = solve_for_nonce_block( - self, - nonce_start, - nonce_end, - block_bytes, - block_difficulty, - self.limit, - block_number, - ) - if solution is not None: - self.solution_queue.put(solution) - - try: - # Send time - self.finished_queue.put_nowait(self.proc_num) - except Full: - pass - - nonce_start = random.randint(0, nonce_limit) - nonce_start = nonce_start % nonce_limit - nonce_end = nonce_start + self.update_interval - - -class CUDASolver(SolverBase): - dev_id: int - TPB: int - - def __init__( - self, - proc_num, - num_proc, - update_interval, - finished_queue, - solution_queue, - stopEvent, - curr_block, - curr_block_num, - curr_diff, - check_block, - limit, - dev_id: int, - TPB: int, - ): - super().__init__( - proc_num, - num_proc, - update_interval, - finished_queue, - solution_queue, - stopEvent, - curr_block, - curr_block_num, - curr_diff, - check_block, - limit, - ) - self.dev_id = dev_id - self.TPB = TPB - - def run(self): - block_number: int = 0 # dummy value - block_bytes: bytes = b"0" * 32 # dummy value - block_difficulty: int = int(math.pow(2, 64)) - 1 # dummy value - nonce_limit = int(math.pow(2, 64)) - 1 # U64MAX - - # Start at random nonce - nonce_start = random.randint(0, nonce_limit) - while not self.stopEvent.is_set(): - if self.newBlockEvent.is_set(): - with self.check_block: - block_number = self.curr_block_num.value - block_bytes = bytes(self.curr_block) - block_difficulty = registration_diff_unpack(self.curr_diff) - - self.newBlockEvent.clear() - - # Do a block of nonces - solution = solve_for_nonce_block_cuda( - self, - nonce_start, - self.update_interval, - block_bytes, - block_difficulty, - self.limit, - block_number, - self.dev_id, - self.TPB, - ) - if solution is not None: - self.solution_queue.put(solution) - - try: - # Signal that a nonce_block was finished using queue - # send our proc_num - self.finished_queue.put(self.proc_num) - except Full: - pass - - # increase nonce by number of nonces processed - nonce_start += self.update_interval * self.TPB - nonce_start = nonce_start % nonce_limit - - -def solve_for_nonce_block_cuda( - solver: CUDASolver, - nonce_start: int, - update_interval: int, - block_bytes: bytes, - difficulty: int, - limit: int, - block_number: int, - dev_id: int, - TPB: int, -) -> Optional[POWSolution]: - """Tries to solve the POW on a CUDA device for a block of nonces (nonce_start, nonce_start + update_interval * TPB""" - solution, seal = solve_cuda( - nonce_start, update_interval, TPB, block_bytes, difficulty, limit, dev_id - ) - - if solution != -1: - # Check if solution is valid (i.e. not -1) - return POWSolution(solution, block_number, difficulty, seal) - - return None - - -def solve_for_nonce_block( - solver: Solver, - nonce_start: int, - nonce_end: int, - block_bytes: bytes, - difficulty: int, - limit: int, - block_number: int, -) -> Optional[POWSolution]: - """Tries to solve the POW for a block of nonces (nonce_start, nonce_end)""" - for nonce in range(nonce_start, nonce_end): - # Create seal. - nonce_bytes = binascii.hexlify(nonce.to_bytes(8, "little")) - pre_seal = nonce_bytes + block_bytes - seal_sh256 = hashlib.sha256(bytearray(hex_bytes_to_u8_list(pre_seal))).digest() - kec = keccak.new(digest_bits=256) - seal = kec.update(seal_sh256).digest() - seal_number = int.from_bytes(seal, "big") - - # Check if seal meets difficulty - product = seal_number * difficulty - if product < limit: - # Found a solution, save it. - return POWSolution(nonce, block_number, difficulty, seal) - - return None - - -def registration_diff_unpack(packed_diff: multiprocessing.Array) -> int: - """Unpacks the packed two 32-bit integers into one 64-bit integer. Little endian.""" - return int(packed_diff[0] << 32 | packed_diff[1]) - - -def registration_diff_pack(diff: int, packed_diff: multiprocessing.Array): - """Packs the difficulty into two 32-bit integers. Little endian.""" - packed_diff[0] = diff >> 32 - packed_diff[1] = diff & 0xFFFFFFFF # low 32 bits - - -def update_curr_block( - curr_diff: multiprocessing.Array, - curr_block: multiprocessing.Array, - curr_block_num: multiprocessing.Value, - block_number: int, - block_bytes: bytes, - diff: int, - lock: multiprocessing.Lock, -): - with lock: - curr_block_num.value = block_number - for i in range(64): - curr_block[i] = block_bytes[i] - registration_diff_pack(diff, curr_diff) - - -def get_cpu_count(): - try: - return len(os.sched_getaffinity(0)) - except AttributeError: - # OSX does not have sched_getaffinity - return os.cpu_count() - - -@dataclass -class RegistrationStatistics: - """Statistics for a registration.""" - - time_spent_total: float - rounds_total: int - time_average: float - time_spent: float - hash_rate_perpetual: float - hash_rate: float - difficulty: int - block_number: int - block_hash: bytes - - -class RegistrationStatisticsLogger: - """Logs statistics for a registration.""" - - console: rich_console.Console - status: Optional[rich_status.Status] - - def __init__( - self, console: rich_console.Console, output_in_place: bool = True - ) -> None: - self.console = console - - if output_in_place: - self.status = self.console.status("Solving") - else: - self.status = None - - def start(self) -> None: - if self.status is not None: - self.status.start() - - def stop(self) -> None: - if self.status is not None: - self.status.stop() - - def get_status_message( - cls, stats: RegistrationStatistics, verbose: bool = False - ) -> str: - message = ( - "Solving\n" - + f"Time Spent (total): [bold white]{timedelta(seconds=stats.time_spent_total)}[/bold white]\n" - + ( - f"Time Spent This Round: {timedelta(seconds=stats.time_spent)}\n" - + f"Time Spent Average: {timedelta(seconds=stats.time_average)}\n" - if verbose - else "" - ) - + f"Registration Difficulty: [bold white]{millify(stats.difficulty)}[/bold white]\n" - + f"Iters (Inst/Perp): [bold white]{get_human_readable(stats.hash_rate, 'H')}/s / " - + f"{get_human_readable(stats.hash_rate_perpetual, 'H')}/s[/bold white]\n" - + f"Block Number: [bold white]{stats.block_number}[/bold white]\n" - + f"Block Hash: [bold white]{stats.block_hash.encode('utf-8')}[/bold white]\n" - ) - return message - - def update(self, stats: RegistrationStatistics, verbose: bool = False) -> None: - if self.status is not None: - self.status.update(self.get_status_message(stats, verbose=verbose)) - else: - self.console.log(self.get_status_message(stats, verbose=verbose)) - - -def solve_for_difficulty_fast( - subtensor: "bittensor.subtensor", - wallet, - output_in_place: bool = True, - num_processes: Optional[int] = None, - update_interval: Optional[int] = None, - n_samples: int = 10, - alpha_: float = 0.80, - log_verbose: bool = False, -) -> Optional[POWSolution]: - """ - Solves the POW for registration using multiprocessing. - Args: - subtensor - Subtensor to connect to for block information and to submit. - wallet: - Wallet to use for registration. - output_in_place: bool - If true, prints the status in place. Otherwise, prints the status on a new line. - num_processes: int - Number of processes to use. - update_interval: int - Number of nonces to solve before updating block information. - n_samples: int - The number of samples of the hash_rate to keep for the EWMA - alpha_: float - The alpha for the EWMA for the hash_rate calculation - log_verbose: bool - If true, prints more verbose logging of the registration metrics. - Note: The hash rate is calculated as an exponentially weighted moving average in order to make the measure more robust. - Note: - - We can also modify the update interval to do smaller blocks of work, - while still updating the block information after a different number of nonces, - to increase the transparency of the process while still keeping the speed. - """ - if num_processes == None: - # get the number of allowed processes for this process - num_processes = min(1, get_cpu_count()) - - if update_interval is None: - update_interval = 50_000 - - limit = int(math.pow(2, 256)) - 1 - - curr_block = multiprocessing.Array("h", 64, lock=True) # byte array - curr_block_num = multiprocessing.Value("i", 0, lock=True) # int - curr_diff = multiprocessing.Array("Q", [0, 0], lock=True) # [high, low] - - # Establish communication queues - ## See the Solver class for more information on the queues. - stopEvent = multiprocessing.Event() - stopEvent.clear() - - solution_queue = multiprocessing.Queue() - finished_queues = [multiprocessing.Queue() for _ in range(num_processes)] - check_block = multiprocessing.Lock() - - # Start consumers - solvers = [ - Solver( - i, - num_processes, - update_interval, - finished_queues[i], - solution_queue, - stopEvent, - curr_block, - curr_block_num, - curr_diff, - check_block, - limit, - ) - for i in range(num_processes) - ] - - # Get first block - block_number = subtensor.get_current_block() - difficulty = subtensor.difficulty - block_hash = subtensor.substrate.get_block_hash(block_number) - while block_hash == None: - block_hash = subtensor.substrate.get_block_hash(block_number) - block_bytes = block_hash.encode("utf-8")[2:] - old_block_number = block_number - # Set to current block - update_curr_block( - curr_diff, - curr_block, - curr_block_num, - block_number, - block_bytes, - difficulty, - check_block, - ) - - # Set new block events for each solver to start at the initial block - for worker in solvers: - worker.newBlockEvent.set() - - for worker in solvers: - worker.start() # start the solver processes - - start_time = time.time() # time that the registration started - time_last = start_time # time that the last work blocks completed - - curr_stats = RegistrationStatistics( - time_spent_total=0.0, - time_average=0.0, - rounds_total=0, - time_spent=0.0, - hash_rate_perpetual=0.0, - hash_rate=0.0, - difficulty=difficulty, - block_number=block_number, - block_hash=block_hash, - ) - - start_time_perpetual = time.time() - - console = bittensor.__console__ - logger = RegistrationStatisticsLogger(console, output_in_place) - logger.start() - - solution = None - - hash_rates = [0] * n_samples # The last n true hash_rates - weights = [alpha_**i for i in range(n_samples)] # weights decay by alpha - - while not subtensor.is_hotkey_registered(hotkey_ss58=wallet.hotkey.ss58_address): - # Wait until a solver finds a solution - try: - solution = solution_queue.get(block=True, timeout=0.25) - if solution is not None: - break - except Empty: - # No solution found, try again - pass - - # check for new block - old_block_number = check_for_newest_block_and_update( - subtensor=subtensor, - old_block_number=old_block_number, - curr_diff=curr_diff, - curr_block=curr_block, - curr_block_num=curr_block_num, - curr_stats=curr_stats, - update_curr_block=update_curr_block, - check_block=check_block, - solvers=solvers, - ) - - num_time = 0 - for finished_queue in finished_queues: - try: - proc_num = finished_queue.get(timeout=0.1) - num_time += 1 - - except Empty: - continue - - time_now = time.time() # get current time - time_since_last = time_now - time_last # get time since last work block(s) - if num_time > 0 and time_since_last > 0.0: - # create EWMA of the hash_rate to make measure more robust - - hash_rate_ = (num_time * update_interval) / time_since_last - hash_rates.append(hash_rate_) - hash_rates.pop(0) # remove the 0th data point - curr_stats.hash_rate = sum( - [hash_rates[i] * weights[i] for i in range(n_samples)] - ) / (sum(weights)) - - # update time last to now - time_last = time_now - - curr_stats.time_average = ( - curr_stats.time_average * curr_stats.rounds_total - + curr_stats.time_spent - ) / (curr_stats.rounds_total + num_time) - curr_stats.rounds_total += num_time - - # Update stats - curr_stats.time_spent = time_since_last - new_time_spent_total = time_now - start_time_perpetual - curr_stats.hash_rate_perpetual = ( - curr_stats.rounds_total * update_interval - ) / new_time_spent_total - curr_stats.time_spent_total = new_time_spent_total - - # Update the logger - logger.update(curr_stats, verbose=log_verbose) - - # exited while, solution contains the nonce or wallet is registered - stopEvent.set() # stop all other processes - logger.stop() - - # terminate and wait for all solvers to exit - terminate_workers_and_wait_for_exit(solvers) - - return solution - - -@backoff.on_exception(backoff.constant, Exception, interval=1, max_tries=3) -def get_block_with_retry(subtensor: "bittensor.subtensor") -> Tuple[int, int, bytes]: - block_number = subtensor.get_current_block() - difficulty = subtensor.difficulty - block_hash = subtensor.substrate.get_block_hash(block_number) - if block_hash is None: - raise Exception( - "Network error. Could not connect to substrate to get block hash" - ) - return block_number, difficulty, block_hash - - -class UsingSpawnStartMethod: - def __init__(self, force: bool = False): - self._old_start_method = None - self._force = force - - def __enter__(self): - self._old_start_method = multiprocessing.get_start_method(allow_none=True) - if self._old_start_method == None: - self._old_start_method = "spawn" # default to spawn - - multiprocessing.set_start_method("spawn", force=self._force) - - def __exit__(self, *args): - # restore the old start method - multiprocessing.set_start_method(self._old_start_method, force=True) - - -def check_for_newest_block_and_update( - subtensor: "bittensor.subtensor", - old_block_number: int, - curr_diff: multiprocessing.Array, - curr_block: multiprocessing.Array, - curr_block_num: multiprocessing.Value, - update_curr_block: Callable, - check_block: "multiprocessing.Lock", - solvers: List[Solver], - curr_stats: RegistrationStatistics, -) -> int: - """ - Checks for a new block and updates the current block information if a new block is found. - Args: - subtensor (:obj:`bittensor.subtensor`, `required`): - The subtensor object to use for getting the current block. - old_block_number (:obj:`int`, `required`): - The old block number to check against. - curr_diff (:obj:`multiprocessing.Array`, `required`): - The current difficulty as a multiprocessing array. - curr_block (:obj:`multiprocessing.Array`, `required`): - Where the current block is stored as a multiprocessing array. - curr_block_num (:obj:`multiprocessing.Value`, `required`): - Where the current block number is stored as a multiprocessing value. - update_curr_block (:obj:`Callable`, `required`): - A function that updates the current block. - check_block (:obj:`multiprocessing.Lock`, `required`): - A mp lock that is used to check for a new block. - solvers (:obj:`List[Solver]`, `required`): - A list of solvers to update the current block for. - curr_stats (:obj:`RegistrationStatistics`, `required`): - The current registration statistics to update. - Returns: - (int) The current block number. - """ - block_number = subtensor.get_current_block() - if block_number != old_block_number: - old_block_number = block_number - # update block information - block_hash = subtensor.substrate.get_block_hash(block_number) - while block_hash == None: - block_hash = subtensor.substrate.get_block_hash(block_number) - block_bytes = block_hash.encode("utf-8")[2:] - difficulty = subtensor.difficulty - - update_curr_block( - curr_diff, - curr_block, - curr_block_num, - block_number, - block_bytes, - difficulty, - check_block, - ) - # Set new block events for each solver - - for worker in solvers: - worker.newBlockEvent.set() - - # update stats - curr_stats.block_number = block_number - curr_stats.block_hash = block_hash - curr_stats.difficulty = difficulty - - return old_block_number - - -def solve_for_difficulty_fast_cuda( - subtensor: "bittensor.subtensor", - wallet: "bittensor.wallet", - output_in_place: bool = True, - update_interval: int = 50_000, - TPB: int = 512, - dev_id: Union[List[int], int] = 0, - n_samples: int = 10, - alpha_: float = 0.80, - log_verbose: bool = False, -) -> Optional[POWSolution]: - """ - Solves the registration fast using CUDA - Args: - subtensor: bittensor.subtensor - The subtensor node to grab blocks - wallet: bittensor.wallet - The wallet to register - output_in_place: bool - If true, prints the output in place, otherwise prints to new lines - update_interval: int - The number of nonces to try before checking for more blocks - TPB: int - The number of threads per block. CUDA param that should match the GPU capability - dev_id: Union[List[int], int] - The CUDA device IDs to execute the registration on, either a single device or a list of devices - n_samples: int - The number of samples of the hash_rate to keep for the EWMA - alpha_: float - The alpha for the EWMA for the hash_rate calculation - log_verbose: bool - If true, prints more verbose logging of the registration metrics. - Note: The hash rate is calculated as an exponentially weighted moving average in order to make the measure more robust. - """ - if isinstance(dev_id, int): - dev_id = [dev_id] - elif dev_id is None: - dev_id = [0] - - if update_interval is None: - update_interval = 50_000 - - if not torch.cuda.is_available(): - raise Exception("CUDA not available") - - limit = int(math.pow(2, 256)) - 1 - - # Set mp start to use spawn so CUDA doesn't complain - with UsingSpawnStartMethod(force=True): - curr_block = multiprocessing.Array("h", 64, lock=True) # byte array - curr_block_num = multiprocessing.Value("i", 0, lock=True) # int - curr_diff = multiprocessing.Array("Q", [0, 0], lock=True) # [high, low] - - ## Create a worker per CUDA device - num_processes = len(dev_id) - - # Establish communication queues - stopEvent = multiprocessing.Event() - stopEvent.clear() - solution_queue = multiprocessing.Queue() - finished_queues = [multiprocessing.Queue() for _ in range(num_processes)] - check_block = multiprocessing.Lock() - - # Start workers - solvers = [ - CUDASolver( - i, - num_processes, - update_interval, - finished_queues[i], - solution_queue, - stopEvent, - curr_block, - curr_block_num, - curr_diff, - check_block, - limit, - dev_id[i], - TPB, - ) - for i in range(num_processes) - ] - - # Get first block - block_number = subtensor.get_current_block() - difficulty = subtensor.difficulty - block_hash = subtensor.substrate.get_block_hash(block_number) - while block_hash == None: - block_hash = subtensor.substrate.get_block_hash(block_number) - block_bytes = block_hash.encode("utf-8")[2:] - old_block_number = block_number - - # Set to current block - update_curr_block( - curr_diff, - curr_block, - curr_block_num, - block_number, - block_bytes, - difficulty, - check_block, - ) - - # Set new block events for each solver to start at the initial block - for worker in solvers: - worker.newBlockEvent.set() - - for worker in solvers: - worker.start() # start the solver processes - - start_time = time.time() # time that the registration started - time_last = start_time # time that the last work blocks completed - - curr_stats = RegistrationStatistics( - time_spent_total=0.0, - time_average=0.0, - rounds_total=0, - time_spent=0.0, - hash_rate_perpetual=0.0, - hash_rate=0.0, # EWMA hash_rate (H/s) - difficulty=difficulty, - block_number=block_number, - block_hash=block_hash, - ) - - start_time_perpetual = time.time() - - console = bittensor.__console__ - logger = RegistrationStatisticsLogger(console, output_in_place) - logger.start() - - hash_rates = [0] * n_samples # The last n true hash_rates - weights = [alpha_**i for i in range(n_samples)] # weights decay by alpha - - solution = None - while not subtensor.is_hotkey_registered( - hotkey_ss58=wallet.hotkey.ss58_address - ): - # Wait until a solver finds a solution - try: - solution = solution_queue.get(block=True, timeout=0.15) - if solution is not None: - break - except Empty: - # No solution found, try again - pass - - # check for new block - old_block_number = check_for_newest_block_and_update( - subtensor=subtensor, - curr_diff=curr_diff, - curr_block=curr_block, - curr_block_num=curr_block_num, - old_block_number=old_block_number, - curr_stats=curr_stats, - update_curr_block=update_curr_block, - check_block=check_block, - solvers=solvers, - ) - - num_time = 0 - # Get times for each solver - for finished_queue in finished_queues: - try: - proc_num = finished_queue.get(timeout=0.1) - num_time += 1 - - except Empty: - continue - - time_now = time.time() # get current time - time_since_last = time_now - time_last # get time since last work block(s) - if num_time > 0 and time_since_last > 0.0: - # create EWMA of the hash_rate to make measure more robust - - hash_rate_ = (num_time * TPB * update_interval) / time_since_last - hash_rates.append(hash_rate_) - hash_rates.pop(0) # remove the 0th data point - curr_stats.hash_rate = sum( - [hash_rates[i] * weights[i] for i in range(n_samples)] - ) / (sum(weights)) - - # update time last to now - time_last = time_now - - curr_stats.time_average = ( - curr_stats.time_average * curr_stats.rounds_total - + curr_stats.time_spent - ) / (curr_stats.rounds_total + num_time) - curr_stats.rounds_total += num_time - - # Update stats - curr_stats.time_spent = time_since_last - new_time_spent_total = time_now - start_time_perpetual - curr_stats.hash_rate_perpetual = ( - curr_stats.rounds_total * (TPB * update_interval) - ) / new_time_spent_total - curr_stats.time_spent_total = new_time_spent_total - - # Update the logger - logger.update(curr_stats, verbose=log_verbose) - - # exited while, found_solution contains the nonce or wallet is registered - - stopEvent.set() # stop all other processes - logger.stop() - - # terminate and wait for all solvers to exit - terminate_workers_and_wait_for_exit(solvers) - - return solution - - -def terminate_workers_and_wait_for_exit(workers: List[multiprocessing.Process]) -> None: - for worker in workers: - worker.terminate() - worker.join() - - -def create_pow( - subtensor, - wallet, - output_in_place: bool = True, - cuda: bool = False, - dev_id: Union[List[int], int] = 0, - tpb: int = 256, - num_processes: int = None, - update_interval: int = None, - log_verbose: bool = False, -) -> Optional[Dict[str, Any]]: - if cuda: - solution: POWSolution = solve_for_difficulty_fast_cuda( - subtensor, - wallet, - output_in_place=output_in_place, - dev_id=dev_id, - TPB=tpb, - update_interval=update_interval, - log_verbose=log_verbose, - ) - else: - solution: POWSolution = solve_for_difficulty_fast( - subtensor, - wallet, - output_in_place=output_in_place, - num_processes=num_processes, - update_interval=update_interval, - log_verbose=log_verbose, - ) - - return ( - None - if solution is None - else { - "nonce": solution.nonce, - "difficulty": solution.difficulty, - "block_number": solution.block_number, - "work": binascii.hexlify(solution.seal), - } - )