KOS & CLI (kaspanet#236)

* wip - generator change & fees

* wip - propagate generator to send() and estimate()

* yield_executor in account send()

* Add page size and memory usage in bytes to perf_monitor

This commit adds `page_size` as a parameter in several function definitions in `builder.rs`. It also adds counters to track `resident_set_size_bytes` and `virtual_memory_size_bytes` in `counters.rs`. These changes are designed to enable accurate memory monitoring and provide additional details about memory usage. The new `page_size` parameter allows us to associate memory usage with the page size, which will be helpful for monitoring and troubleshooting.

The `page_size` crate has also been added to `Cargo.toml` to manage the retrieval of system's memory page size. This addition is part of the effort to accurately track and report memory usage in bytes.

* Add performance monitoring to RPC

In this commit, we have integrated the performance monitoring module into the RPC layer of our application. The PerfMonitor module is now added as a dependency in various Cargo.toml files, and methods related to performance monitoring have been added/updated in several .rs files. This allows us to collect various metrics related to our application's performance, including memory usage, CPU usage, disc I/O, etc. which will help in evaluating our software's efficiency. We chose to collect these specific metrics as they give a comprehensive overview of the software's performance and can help us detect potential bottlenecks or issues. We have also updated the ProcessMetrics structure to include these new metrics.

* wip - metrics

* is_standard_output_amount_dust() updated using static/const values

* rename get_connection_info to get_server_info

* wip - kos metrics

* wip - metrics

* wip - tx generator summary struct

* clippy

* wip - metrics (add 2hrs span)

* wip - fix wallet naming during creation

* cleanup

* wip - graph duration (cleanup)

* wip - help now uses term().help()

* wip - general refactoring
kos cli (wizards), rearrange cli commands, usage of term().help() where appropriate, metrics (testing), misc wallet operations bugfixes.

* wip - account ident renaming

* wip - utxo groupting for tx

* wip - fix node sync monito

* fix "no transactions found" error

* disable background console

* cli - fix commit() during account rename

* mutation lock issue

* cli - fix account selection

* rearrange mutexes in mass calculator

* working on tx generator

* wip - generator signer

* sign_with_multiple_v2, tx generator wip

* wip - cli rpc handler

* update metrics to use a published workflow-perf-monitor fork of the perf-monitor crate

* Remove "page_size" and related counters from "perf_monitor"

The "page_size" attribute and its related attributes "resident_set_size_bytes" and "virutal_memory_size_bytes" have been removed from the performance monitor (perf_monitor) module. These counters were found to be unnecessary and their removal improves the simplicity and readability of the module. This change has also led to the removal of the "page_size" package from the dependencies.

* perf-monitor - rename resident_set_size_bytes to resident_set_size

* wip: tx generator

* cli - monitor command

* cli - disable default tracking on mute

* generator docs

* cli - transfer, monitor, generator summary, cleanup

* wip - isolate consensus WASM primitives under a separate module

* native cli - cleanup logs

* cli - refactor help output to use help() helper that helps to output help

* cli - cleanup, parsing helper functions

* wip - address derivation scan fn (for legacy accounts)

* kos - miner throttle setting

* wip - legacy account scans

* wip - imports

* tx mass calculator for WASM

* fix main declaration in kaspa-cli

* wip - wallet imports

* metrics - basic tps indicator

* wallet - switch account names to Option<String>

* wip - metrics retention

* fix filling virtual memory size

* wallet - utxo processor - fix misc issues in utxo processor handling; extend pending transactions; fix issues in utxo iterator; improve event formatting;

* wallet - fix network_id reference; cleanup

* add RPC API versioning provision (handled by get-server-info)

* cli - misc output formatting + select_private_key() handler

* wip - kos README

* kos - metrics toolbar styling

* cleanup

* cleanup

* cleanup

* cleanup

* cleanup and comments

* refactor tx storage to use outgoing transactions, refactor metrics data submission to use raw f64 values, history list vs details; propagate additional data through PendingTransaction and GeneratorSummary.

* cli - cleanup details cmd output

* cli - cleanup account selection output

* fix storage collision between outgoing tx record and change UTXO

* guide and general cleanup

* wallet storage - version

* cleanup

* kos - guide

* kos - README

* main workspace README update

* cleanup

* published workflow-rs 0.4.0

* kos - improve autoconnect

* wallet framework event docs

* cli - fix typo

* kos - fix set_duration error handler

* kos - cleanup logs

* fix lints

* update check script to use --tests and --benches

* fix wallet storage unit test

* js-api - initial Generator harness

* js-api - JsValue for UtxoEntryRference conversion

* js-api - estimateTransactions

* js-api - fix Keypair::toAddress()

* add build script argument propagation

* wallet - add basic user input checks

* WIP: address range with keys

* Add test for transaction bincode serialization

Added a new test function, `test_transaction_bincode`, in `consensus/core/src/tx.rs`. The test case covers the serialization of a transaction structure into bincode and subsequent deserialization. This is beneficial for ensuring the correctness and consistency of the serialization and deserialization processes within our blockchain consensus mechanism.

* Refactor Hash serialization and deserialization

Updated how Hash is serialized and deserialized in crypto/hashes/src/lib.rs.

* Add serde_bytes module for handling byte arrays

This commit introduces a new module called `serde_bytes`. The new module contains two inner modules: `ser` and `de`.
These are for Serde's `Serialize` and `Deserialize`traits respectively. The `ser` module has a `serialize` function for
human-readable and non-human-readable versions of byte arrays. The `de` module essentially does the opposite and contains
a function to deserialize the byte arrays. They both utilize faster_hex library for efficient conversion. This was done
to abstract away byte array serialization/deserialization logic to a separate unit of the project, enhancing modularity
and code organization.

* Improve error handling in serde_bytes implementation

The trait Deserialize in utils/src/serde_bytes/de.rs has been updated to use TryFrom rather than From. The implementation has been updated accordingly.

* Update serde serialization and remove unnecessary dependencies

use utils::serde_bytes instead of external crate serde_bytes

* tmp

* add forceful shutdown on 3 quit attempts

* replace all ToString impls with Display trait impl

* "Implement improved serialization and deserialization for fixed-size bytes"

This update implements a more efficient and concise way to serialize and deserialize fixed-size bytes. It replaces previous code with a new approach that leverages macros and generic traits. The changes were made across multiple files including "subnets.rs", "tx.rs" and several in the "serde_bytes_fixed" directory.

The new approach uses the macro "serde_impl_ser_fixed_bytes" and the trait "serialize", which together streamline the process of serialization. For deserialization, the "deserialize" trait and the macro "serde_impl_deser_fixed_bytes" were used.

These new implementations are designed to support both human-readable and binary formats. This allows for more flexibility and efficiency in handling different types of data. In addition, they improve code coherence and maintainability, making the codebase easier to understand and modify in the future.

* "Implement serialization of SubnetworkId with fixed bytes"

SubnetworkId is now serialized using newly imported serde utilities, serde_impl_ser_fixed_bytes and serde_impl_deser_fixed_bytes instead of serde_bytes_fixed attribute.

* Update formatting of deserialization error message

Modified the "expecting" function in serde_bytes_fixed/de.rs to provide clearer error messages when deserializing byte arrays. The error message now includes the expected data type in addition to the array size.

* wip - refactoring JS APIs

* wip - account refactoring

* wasm - refactor UtxoEntryPointReference TryFrom<&JsValue>

* Refactor deserialization of hexadecimal strings

Reduced code redundancy in serde_bytes_fixed/de.rs and serde_bytes/de.rs by introducing a FromHexVisitor struct which handles converting hexadecimal strings to their respective types. This makes the code more maintainable and easier to test. Moreover, implemented FromHex trait to ScriptPublicKey, which allows us to use our new visitor in deserialization of ScriptPublicKey.

* Update deserialization for fixed-size byte arrays

Fixed-size byte arrays now should implement AsRef<[u8; N]> in addition to FromHex and From<[u8; N]> to fit the constraints. This modification is found in the kaspa-utils library.
This also extends the functionality of the Deserialize trait in kaspa-utils which will now support the deserialization of byte arrays of a fixed size. By doing so, the code maintaining the 'serde_impl_deser_fixed_bytes' and 'serde_impl_ser_fixed_bytes' macros to implement serialization and deserialization for byte arrays of a certain size is simplified. The method works by referencing the byte array as the fixed-size array which makes it possible to use a reference implementation instead of the usual implementation.
Added new Deserialize trait implementation for byte arrays of length 0 to 32 and replaced fixed_bytes with fixed_bytes_ref for module hashes and tx in crypto and consensus/core respectively.
Modified the implementation of the Serialize trait for types that can be referenced as a fixed-sized byte array which is now possible for byte arrays from a length of 0 to 32.
An extended set of examples demonstrating how to use the proposed serialization/deserialization has been included.
The purpose of these changes is to simplify the implementation and usage of serialization/deserialization for byte arrays of a fixed size, and to extend this functionality to a larger range of possible array sizes.

* WIP: gen 0 address manager

* wip - address refactoring

* wip - small utils snippet for argument parsing

* wasm utils script, minor fix for network type

* js-api - generator example

* js-api estimate

* Modified utils.js to allow for more customisation, fixed issues.
Updated simple-transaction.js to use `parseArgs`.

* bugfix potentially undefined addressArg

* update utils to use 'node:util' and use address as destination address

* update rpc.js to use parseArgs

* addresses and hd-wallet

* modify header js

* formatting

* refactor state.js

* refactor storage.js

* refactor test.js

* refactor tx-create.js and introduce guard issynced

* refactor tx-script-sign.js and add guard rpc synced

* network type not used

* refactor tx-send.js and add guard rpc is synced

* refactor utxo-split.js and add guard rpc synced

* add guard to simple-transaction.js

* storage use networktype from args

* rpc - implement get_server_info_call and get_sync_status_call for gRPC proxy

* update Cargo.lock following merge

* Add end-to-end demo

Testing both `createTransaction` and `createTransactions`

* Simplify simple-transactions

* Move mining references

* unit tests: gen0 kaspatest addresses

* wip - runtime account refactoring

* wip - account refactoring

* wallet decrypt error message

* WIP: storage structure update

* wip - account refactoring

* updated demo, tx-create with more comments and use rpc parse url/args
added comments to utils

* a better rpc parse url for tx-script-sign tx-send and utxo-split
updated utils to have a better help message

* dont include null

* dont directly use paymount output

* wip - account refactoring (metadata storage, import scaffolding, account derivation indexes)

* utxo processing settings

* wallet - update stored metadata

* clippy

* Add WASM serialization/deserialization tests and improve error handling

This commit adds unit tests for WASM serialization and deserialization in 'wasm.rs' and also updates relevant dependencies in 'Cargo.toml'.

In addition, this commit enhances error handling in 'lib.rs' by making 'TryFrom<&str> for Version' fallible, meaning it can return an 'AddressError' if an invalid string is converted. This should improve robustness and ease debugging efforts.

Finally, this commit modifies the Deserializer implementation for 'Address' to handle more input data types and give more detailed error messages. This should enhance error handling capability and improve user/dev experience.

* Move test cases from rpc/wrpc/client to crypto/addresses

The test cases related to the functionality of Address in wasm (WebAssembly) were moved from rpc/wrpc/client to the crypto/addresses module. This move improves the modularity and coherence of the tests by placing them directly in the module they're testing.

Additionally, dependencies related to these test cases were added to crypto/addresses module and removed from rpc/wrpc/client, further improving the modular structure of the project.

* Configure wasm tests to run in browser

A line was added to make the WebAssembly tests run in a browser context. This is implemented using the `wasm_bindgen_test_configure!` macro from the `wasm_bindgen_test` crate. This change is necessary to ensure accuracy of WebAssembly functionality in the browser environment.

* Refactor AddressVisitor to simplify default implementation

The Default implementation for AddressVisitor has been condensed for simplification. Instead of manually implementing the Default trait with a function, the built-in derive annotation is used. This refactoring makes code cleaner and easier to understand.

* Remove run_in_browser configuration in wasm test

The 'run_in_browser' configuration that was previously set up in the wasm_bindgen_test has been removed as it is unnecessary. Additionally, an assertion was added to check the type of JavaScript value generated upon serializing Address, to ensure type consistency across serialization and deserialization.

* Add assert for object type in lib.rs

A new assertion has been added in the lib.rs file to ensure that the object type is correctly set to "object".

* wallet core - address scan for keypair and resident addresss

* WIP: account storage structure

* wip - refactoring async

* wip - remove async from some derivation trait fns

* progress working on parse host function

* parse host with tests

* bugfix only port and added more tests

* added docs

* Small changes to parse

* update parse_url to use parse_host function

* add brackets to ipv6 and remove previous impl for parse_url

* use ok or else

* back to .map(Ok)

* parse host to check server if valid input

* wip - migrating parts of the JS SDK into kaspa-consensus-wasm

* wip - cleanup + migrate wasm traits to extensions

* cleanup

* wasm - rename modules

* wip - cleanup

* Update wasm handling in lib.rs

Added special handling for wasm target architecture in lib.rs. The changes include specifying different formats for the formatter, new methods to handle different types of input(particularly: conversion from pointer to Address struct) and added new tests. These changes were necessary to ensure compatibility with the wasm32 architecture

* Update wasm-bindgen-test and web-sys dependency scope

Moved wasm-bindgen-test and web-sys dependencies from specific versions to workspace in crypto/addresses/Cargo.toml file. Also, added wasm-bindgen-test to the parent Cargo.toml. This change was necessary to ensure all dependencies are fetched from the workspace, creating a unified build and test environment.

* Remove unnecessary whitespace in wasm32 function

Whitespace on line 379 between the #[cfg] attribute and the visit_f32 function definition was removed. The change was made to clean up the code and maintain consistency with the rest of the codebase in crypto/addresses/src/lib.rs.

* adress - post-merge fixes and cleanup

* "Refactor ScriptPublicKey implementation in tx.rs"

Moved the implementation of ScriptPublicKey from tx.rs to a new file script_public_key.rs to make the structure of the codebase more organized. Also updated the tx.rs file to import the ScriptPublicKey related types from this new file.

This will improve the readability of the tx.rs file and make it easier for developers to navigate through the codebase. Future improvements and changes related to ScriptPublicKey will now be done in script_public_key.rs, thus making these changes more isolated and easier to manage.

* Add wasm support for ScriptPublicKey

Added WebAssembly (wasm) support to the ScriptPublicKey module in the consensus core, and updated dependencies accordingly. This change allows for ScriptPublicKey usage within a wasm context. Additionally, updated dependencies in Cargo.toml and Cargo.lock to reflect changes needed for wasm support. This improvement is essential to expanding the consensus core's platform interoperability.

The ScriptPublicKeyVisitor struct was added to the src/tx/script_public_key.rs file to support the wasm target. Also, the new FromHexVisitor and TryFrom traits are implemented to transform Value into u16 and Vec<u8> respectively. This is necessary for conversion between numeric and text representations. Wasm tests were also added to ensure proper working of the updates.

The consensus core's interoperability and functionality are improved with this update. Therefore, it expands the platform's usage potential. This is a crucial update for working with wasm environments, especially for web-based projects.

* wasm - expose UtxoProcessor and UtxoContext

* preparing merge into kos PR

* add /test script

* wasm - fix duplics wasm symbol error

* cli - different prompt on shutdown

* kos - update terminal for clipboard copy/paste fix on windows

* wallet core - utxo recovery period

* rustdoc

* wallet storage - stored data version check

* kos 0.0.2

* Carog.lock update

* kos - rename kaspa miner binary during redistributable build

* fix for terminal unicode handling issues

* Common-use case impl and move addresses to main

* WIP: Start implementing a PublicKey class for binding

* Implement toAddress of compressed public key

* Export JSPublicKey from rust as PublicKey in js

* Implement x-only JS pubkey support

* Full DER Public Key support

* wasm - interfaces and prv keydata

* wasm - updates for the keypair merge

* wasm - cleanup, update keypair to use XOnlyPublicKey instead of &serialize[1..]

* wrpc - websocket defaults (tungstenite update)

* workflow-rs 0.6.0

* wip - update JavaScript demo scripts for the latest WASM API changes

* wallet - fix unit tests

* add integration tests and improve /test script

* wasm - UtxoProcessor and UtxoContext APIs, improving demos

* code cleanup

* /test - fix pushd arguments

* wasm - utxo context support for wasm tx generator

* wip - js examples, padding for fees

* cleanup (addressing consensus-related github review topics)

* github review issues (phase II)

* github review (phase 3)

* fix /test script that was failing due to wasm-pack warning (wasm-pack issue)

---------

Co-authored-by: max143672 <biryukov.maxim.vladimirovich@yandex.ru>
Co-authored-by: Surinder Singh Matoo <surinder83singh@gmail.com>
Co-authored-by: Maxim <59533214+biryukovmaxim@users.noreply.github.com>
Co-authored-by: Dennis Kievits <denkievits@gmail.com>
Co-authored-by: coderofstuff <114628839+coderofstuff@users.noreply.github.com>

.gitignore

@@ -7,4 +7,6 @@ web-root
 /testing/integration/testdata/dags_for_json_tests/goref-mainnet
 /testing/integration/testdata/dags_for_json_tests/goref-1.6M-tx-10K-blocks
 analyzer-target
-.DS_Store
+.DS_Store
+*.code-workspace
+/setup