From e02c9c6662823c1e8f04e5c921f544d7031d43f2 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Sun, 1 Sep 2019 21:48:23 -0400 Subject: [PATCH 01/61] Create 0000-online-transacting-via-tor.md --- text/0000-online-transacting-via-tor.md | 40 +++++++++++++++++++++++++ 1 file changed, 40 insertions(+) create mode 100644 text/0000-online-transacting-via-tor.md diff --git a/text/0000-online-transacting-via-tor.md b/text/0000-online-transacting-via-tor.md new file mode 100644 index 0000000..bc5d7d7 --- /dev/null +++ b/text/0000-online-transacting-via-tor.md @@ -0,0 +1,40 @@ +# 0000-online-transacting-via-tor
+Title: online-transacting-via-tor
+Authors: DavidBurkett
+Start date: September 1, 2019
+ +# Summary +Describes a standardized addressing and communication protocol for building Grin transactions. +Wallets will attempt to connect to the recipient directly to build transactions over a TOR hidden service similar to how http(s) transaction building works today. +If the recipient is not online, or cannot be contacted directly, users have the option of falling back to an “offline” communication system where slates are encrypted and stored, for a limited period of time, by a subset(TBD) of nodes known as “relay nodes” (described in a future RFC). + +# Motivation +Grin is unique in that it requires the sender and receiver to interact in order to transact. This presents a lot of unique challenges that most coins don’t have to deal with. There are a number of different incompatible standards for sending and receiving, resulting in confusion and headaches for many users. The hope is that the addressing mechanism described here will become the new default method for sending and receiving, deprecating several less secure and less private methods in the process. + +# Description +## Community-level explanation +From an end-user perspective, there should no longer be a need to configure routers and firewalls to receive grins. Sending and receiving should feel like any other cryptocurrency, where a simple encoded address is all you need to share before receiving. No firewall or router configuration should be necessary. + +## Reference-level explanation +### Addressing +Onion addresses for TOR hidden services are generated from an ed25519 public key (32 bytes), and include a checksum and a version[1]. This provides an equivalent level of security as bitcoin addresses, and can be ephemeral or permanent, depending on the user’s needs. Grin addresses should be generated in the same way as TOR addresses, although we have the flexibility to decide whether we would like to encode them using base32 (like TOR), base58 (like Bitcoin), or even a far superior encoding based on emojis. +Although ed25519 is a different curve than used by the grin protocol, we can still use our HD wallets to generate deterministic ed25519 public keys (and therefore Grin addresses). We can decide to generate them from a special keychain path, or even using a different master seed [2], along with a KDF for converting those HD child keys to ed25519-compatible private keys. + +### TOR Hidden Services +TOR hidden services can be used to directly serve the existing foreign APIs. When configuring TOR (whether bundled with grin, or installed separately), you would just publish a hidden service and configure TOR to forward all traffic to port 3420. This means we can continue supporting http(s) sending/receiving with no disruption, though it’s advisable to avoid sending directly over http(s) asap.In future versions of Grin, we can stop allowing non-local connections to the foreign wallet APIs. + +# Drawbacks +Requires users to setup and configure TOR, or bundle it with Grin, which could be non-trivial, and could conflict with locally installed/running versions. + +# Unresolved questions +* How should we encode addresses? base32, base58, emoji, etc. +* How often should addresses change? Should users manually request a new address, or should they auto expire? Should we support multiple? + +# Future possibilities +The changes in this RFC lead the way for: +* Payment proofs +* Offline transacting via SBBS/Grinbox-style relay system. See “Offline Transactions” RFC + +# References +[1] https://github.com/torproject/torspec/blob/87698dc1c0fa4cf2186f180a636fc7ad1c5fb5fd/rend-spec-v3.txt#L2059-L2081 +[2] https://github.com/mimblewimble/grin/blob/d918c5fe84e859290c9d09f5cfc167ed41d27bff/keychain/src/extkey_bip32.rs#L122 From 9d01bc5ee0b209df726f45165f4aa2f0c0169af5 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Mon, 2 Sep 2019 04:21:08 -0400 Subject: [PATCH 02/61] Update 0000-online-transacting-via-tor.md --- text/0000-online-transacting-via-tor.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/text/0000-online-transacting-via-tor.md b/text/0000-online-transacting-via-tor.md index bc5d7d7..70a55e9 100644 --- a/text/0000-online-transacting-via-tor.md +++ b/text/0000-online-transacting-via-tor.md @@ -4,8 +4,8 @@ Authors: DavidBurkett
Start date: September 1, 2019
# Summary -Describes a standardized addressing and communication protocol for building Grin transactions. -Wallets will attempt to connect to the recipient directly to build transactions over a TOR hidden service similar to how http(s) transaction building works today. +Describes a standardized addressing and communication protocol for building Grin transactions.

+Wallets will attempt to connect to the recipient directly to build transactions over a TOR hidden service similar to how http(s) transaction building works today.

If the recipient is not online, or cannot be contacted directly, users have the option of falling back to an “offline” communication system where slates are encrypted and stored, for a limited period of time, by a subset(TBD) of nodes known as “relay nodes” (described in a future RFC). # Motivation @@ -17,7 +17,7 @@ From an end-user perspective, there should no longer be a need to configure rout ## Reference-level explanation ### Addressing -Onion addresses for TOR hidden services are generated from an ed25519 public key (32 bytes), and include a checksum and a version[1]. This provides an equivalent level of security as bitcoin addresses, and can be ephemeral or permanent, depending on the user’s needs. Grin addresses should be generated in the same way as TOR addresses, although we have the flexibility to decide whether we would like to encode them using base32 (like TOR), base58 (like Bitcoin), or even a far superior encoding based on emojis. +Onion addresses for TOR hidden services are generated from an ed25519 public key (32 bytes), and include a checksum and a version[1]. This provides an equivalent level of security as bitcoin addresses, and can be ephemeral or permanent, depending on the user’s needs. Grin addresses should be generated in the same way as TOR addresses, although we have the flexibility to decide whether we would like to encode them using base32 (like TOR), base58 (like Bitcoin), or even a far superior encoding based on emojis.

Although ed25519 is a different curve than used by the grin protocol, we can still use our HD wallets to generate deterministic ed25519 public keys (and therefore Grin addresses). We can decide to generate them from a special keychain path, or even using a different master seed [2], along with a KDF for converting those HD child keys to ed25519-compatible private keys. ### TOR Hidden Services From 1623aaaee36257579e116266d48861a0096a25a6 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Mon, 2 Sep 2019 04:21:20 -0400 Subject: [PATCH 03/61] Update 0000-online-transacting-via-tor.md --- text/0000-online-transacting-via-tor.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0000-online-transacting-via-tor.md b/text/0000-online-transacting-via-tor.md index 70a55e9..cdcfcd5 100644 --- a/text/0000-online-transacting-via-tor.md +++ b/text/0000-online-transacting-via-tor.md @@ -36,5 +36,5 @@ The changes in this RFC lead the way for: * Offline transacting via SBBS/Grinbox-style relay system. See “Offline Transactions” RFC # References -[1] https://github.com/torproject/torspec/blob/87698dc1c0fa4cf2186f180a636fc7ad1c5fb5fd/rend-spec-v3.txt#L2059-L2081 +[1] https://github.com/torproject/torspec/blob/87698dc1c0fa4cf2186f180a636fc7ad1c5fb5fd/rend-spec-v3.txt#L2059-L2081
[2] https://github.com/mimblewimble/grin/blob/d918c5fe84e859290c9d09f5cfc167ed41d27bff/keychain/src/extkey_bip32.rs#L122 From 13c5cd262ec175f2211d0f92a989037a409f37a3 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Wed, 4 Sep 2019 05:17:32 -0400 Subject: [PATCH 04/61] Update 0000-online-transacting-via-tor.md --- text/0000-online-transacting-via-tor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-online-transacting-via-tor.md b/text/0000-online-transacting-via-tor.md index cdcfcd5..4ac2d09 100644 --- a/text/0000-online-transacting-via-tor.md +++ b/text/0000-online-transacting-via-tor.md @@ -6,7 +6,7 @@ Start date: September 1, 2019
# Summary Describes a standardized addressing and communication protocol for building Grin transactions.

Wallets will attempt to connect to the recipient directly to build transactions over a TOR hidden service similar to how http(s) transaction building works today.

-If the recipient is not online, or cannot be contacted directly, users have the option of falling back to an “offline” communication system where slates are encrypted and stored, for a limited period of time, by a subset(TBD) of nodes known as “relay nodes” (described in a future RFC). +If the recipient is not online, or cannot be contacted directly, users have the option of falling back to an asynchronous communication system where slates are encrypted and stored, for a limited period of time, by “relay nodes” (described in 0000-asynchronous-transacting-via-relays). # Motivation Grin is unique in that it requires the sender and receiver to interact in order to transact. This presents a lot of unique challenges that most coins don’t have to deal with. There are a number of different incompatible standards for sending and receiving, resulting in confusion and headaches for many users. The hope is that the addressing mechanism described here will become the new default method for sending and receiving, deprecating several less secure and less private methods in the process. @@ -33,7 +33,7 @@ Requires users to setup and configure TOR, or bundle it with Grin, which could b # Future possibilities The changes in this RFC lead the way for: * Payment proofs -* Offline transacting via SBBS/Grinbox-style relay system. See “Offline Transactions” RFC +* Offline transacting via SBBS/Grinbox-style relay system. See 0000-asynchronous-transacting-via-relays RFC # References [1] https://github.com/torproject/torspec/blob/87698dc1c0fa4cf2186f180a636fc7ad1c5fb5fd/rend-spec-v3.txt#L2059-L2081
From 287668b4c591c3e808dbdd4b85b685641005797b Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Tue, 1 Oct 2019 15:34:15 -0400 Subject: [PATCH 05/61] Node API V2 --- text/0000-node-api-v2.md | 203 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 text/0000-node-api-v2.md diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md new file mode 100644 index 0000000..b841aa5 --- /dev/null +++ b/text/0000-node-api-v2.md @@ -0,0 +1,203 @@ + +- Title: Node API V2 +- Authors: [Quentin Le Sceller](mailto:q.lesceller@gmail.com) +- Start date: September 30th, 2019 +- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) +- Tracking issue: [Edit if merged with link to tracking github issue] + +--- + +# Summary + +[summary]: #summary + +Create a v2 JSON-RPC API for the Node API. + +# Motivation + +[motivation]: #motivation + +The current Node API (referred here as v1 API) is a REST API. This API while simple is: + +- Documented manually (current documentation is available [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md). +- Contains call with heterogenous args such as `?byid=xxx` and `commitment/xxx` which can be confusing and lack some uniformity. +- Uses REST which is bound to HTTP while v2 wallet API uses JSON-RPC. + +The goal of this RFC is to provide a new API with cleaner methods and errors, automatically generated documentation directly on docs.rs and finally stronger basis for future improvements. + +# Community-level explanation + +[community-level-explanation]: #community-level-explanation + +This new API will be particularly useful for wallet developer and should ultimately simplify their work on Grin. Moreover, this RFC does not introduce any breaking changes as the v1 REST API will still be around until completely deprecated. + +# Reference-level explanation + +[reference-level-explanation]: #reference-level-explanation + +## Current Endpoints with the v1 API + +While the endpoints are documented in details [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md), here is an overview of the current REST API Endpoints of the V1 API. + +```JSON +[ + "get blocks", + "get headers", + "get chain", + "post chain/compact", + "get chain/validate", + "get chain/kernels/xxx?min_height=yyy&max_height=zzz", + "get chain/outputs/byids?id=xxx,yyy,zzz", + "get chain/outputs/byheight?start_height=101&end_height=200", + "get status", + "get txhashset/roots", + "get txhashset/lastoutputs?n=10", + "get txhashset/lastrangeproofs", + "get txhashset/lastkernels", + "get txhashset/outputs?start_index=1&max=100", + "get txhashset/merkleproof?n=1", + "get pool", + "post pool/push_tx", + "post peers/a.b.c.d:p/ban", + "post peers/a.b.c.d:p/unban", + "get peers/all", + "get peers/connected", + "get peers/a.b.c.d", + "get version" +] +``` + +This endpoints can be grouped into 5 categories: + +- miscellaneous endpoints (which contain `status` and `version` endpoints) +- `chain` endpoints (which also contain `blocks`, `chain` and headers` endpoints) +- `peer` endpoints +- `pool` endpoints +- `txhashset` endpoints + +## Proposed Endpoints + +### Miscellaneous endpoints + +#### get_status + +Returns various information about the node, the network and the current sync status. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_status", + "params": [], + "id": 1 +} +{ +"id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "protocol_version": "2", + "user_agent": "MW/Grin 2.x.x", + "connections": "8", + "tip": { + "height": 371553, + "last_block_pushed": "00001d1623db988d7ed10c5b6319360a52f20c89b4710474145806ba0e8455ec", + "prev_block_to_last": "0000029f51bacee81c49a27b4bc9c6c446e03183867c922890f90bb17108d89f", + "total_difficulty": 1127628411943045 + }, + "sync_status": "header_sync", + "sync_info": { + "current_height": 371553, + "highest_height": 0 + } + } + } +} +``` + +#### get_version + +Returns the node version and block header version (used by grin-wallet). + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_version", + "params": [], + "id": 1 +} +{ +"id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "node_version": "2.1.0-beta.2", + "block_header_version": 2 + } + } +} +``` + +### Chain endpoints + +TBD + +### Peer endpoints + +TBD + +### Pool endpoints + +TBD + +### TxHashSet endpoints + +TBD + +## Authentication + +Similarly to the V1 API, the v2 API will use basic auth with the same secret. This token is usually in `grin/main/.api_secret`. + +### Wallet support + +The wallet currently uses the Grin Node API v1 so changes will be needed in the grin-wallet repository to make it compatible with the v2 Node API. + +### Legacy support + +The V1 API will remain active for a time the mode of operation for its REST API will be assumed to work as currently. This setup should allow existing wallets and apps to continue working as-is until a cutoff release for legacy mode is determined. + +### API only + +Note that this RFC does not propose making user-facing changes to the existing CLI (invoked by `grin client`) to invoke these functions. It's expected that the existing cli functionality will be modified to invoke the new API functions. + +# Drawbacks + +[drawbacks]: #drawbacks + +The implementation of this RFC will temporarily introduces some additional code complexity as v1 and v2 node API will need to coexist until a cutoff release is determined. + +# Prior art + +[prior-art]: #prior-art + +This kind of JSON-RPC API is widely used for cryptocurrencies. For instance: + +- [Bitcoin](https://en.bitcoin.it/wiki/API_reference_\(JSON-RPC\)) +- [Ethereum](https://github.com/ethereum/wiki/wiki/JSON-RPC) +- [Monero](https://web.getmonero.org/resources/developer-guides/wallet-rpc.html) +- [Tezos](https://tezos.gitlab.io/master/developer/rpc.html) + +# Future possibilities + +[future-possibilities]: #future-possibilities + +This new API will simplify the deployment of new methods and will drastically simplify the work of developers by providing a clear documentation directly on docs.rs. + +# References + +[references]: #references + +- [JSON-RPC 2.0 Specification](https://www.jsonrpc.org/specification) +- [Bitcoin JSON-RPC Doc](https://en.bitcoin.it/wiki/API_reference_\(JSON-RPC\)) +- [Ethereum JSON-RPC Doc](https://github.com/ethereum/wiki/wiki/JSON-RPC) +- [Monero JSON-RPC Doc](https://web.getmonero.org/resources/developer-guides/wallet-rpc.html) +- [Tezos JSON-RPC Doc](https://tezos.gitlab.io/master/developer/rpc.html) From 53084b3129857711baecace1d6d8798a06330cc6 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 3 Oct 2019 11:33:30 -0400 Subject: [PATCH 06/61] Add get_block method --- text/0000-node-api-v2.md | 106 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 105 insertions(+), 1 deletion(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index b841aa5..63fd489 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -139,7 +139,111 @@ Returns the node version and block header version (used by grin-wallet). ### Chain endpoints -TBD +#### get_block + +Gets block details given either the height, the hash or an unspent output commitment. +Only one parameters is needed. If multiple parameters are provided only the first one in the list is used. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_block", + "params": [374274, null, null], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "header": { + "cuckoo_solution": [ + 1263501, + 14648727, + 42430559, + 58137254, + 68666726, + 72784903, + 101936839, + 104273571, + 123886748, + 131179768, + 155443226, + 162493783, + 164784425, + 167313215, + 169806918, + 183041591, + 184403611, + 210351649, + 215159650, + 239995384, + 240935454, + 257742462, + 280820644, + 300143903, + 303146496, + 311804841, + 341039986, + 354918290, + 363508555, + 377618528, + 396693709, + 397417856, + 399875872, + 413238540, + 413767813, + 432697194, + 436903767, + 447257325, + 453337210, + 459401597, + 496068509, + 511300624 + ], + "edge_bits": 29, + "hash": "000001e16cb374e38c979c353a0aaffbf5b939da7688f69ad99efda6c112ea9b", + "height": 374274, + "kernel_root": "e17920c0e456a6feebf19e24a46f510a85f21cb60e81012f843c00fe2c4cad6e", + "nonce": 4354431877761457166, + "output_root": "1e9daee31b80c6b83573eacfd3048a4af57c614bd36f9acd5fb50fbd236beb16", + "prev_root": "9827b8ffab942e264b6ac81f2b487e3de65e411145c514092ce783df9344fa8a", + "previous": "00001266a73ba6a8032ef8b4d4f5508407ffb1c270c105dac06f4669c17af020", + "range_proof_root": "3491b8c46a3919df637a636ca72824377f89c4967dcfe4857379a4a82b510069", + "secondary_scaling": 571, + "timestamp": "2019-10-03T15:15:35+00:00", + "total_difficulty": 1133438031814173, + "total_kernel_offset": "63315ca0be65c9f6ddf2d3306876caf9f458a01d1a0bf50cc4d3c9b699161958", + "version": 2 + }, + "inputs": [], + "kernels": [ + { + "excess": "08761e9cb1eea5bfcf771d1218b5ec802798d6eecaf75faae50ba3a1997aaef009", + "excess_sig": "971317046c533d21dff3e449cc9380c2be10b0274f70e009aa2453f755239e3299883c09a1785b15a141d89d563cdd59395886c7d63aba9c2b6438575555e2c4", + "features": "Coinbase", + "fee": 0, + "lock_height": 0 + } + ], + "outputs": [ + { + "block_height": 374274, + "commit": "09d33615563ba2d65acc2b295a024337166b9f520122d49730c73e8bfb43017610", + "merkle_proof": "00000000003e6f5e000000000000000f60fe09a7601a519d9be71135404580ad9de0964c70a7619b1731dca2cd8c1ae1dce9f544df671d63ff0e05b58f070cb48e163ca8f44fb4446c9fe1fc9cfef90e4b81e7119e8cf60acb9515363ecaea1ce20d2a8ea2f6f638f79a33a19d0d7b54cfff3daf8d21c243ba4ccd2c0fbda833edfa2506b1b326059d124e0c2e27cda90268e66f2dcc7576efac9ebbb831894d7776c191671c3294c2ca0af23201498a7f5ce98d5440ca24116b40ac98b1c5e38b28c8b560afc4f4684b81ab34f8cf162201040d4779195ba0e4967d1dd8184b579208e9ebebafa2f5004c51f5902a94bf268fd498f0247e8ba1a46efec8d88fa44d5ecb206fbe728ee56c24af36442eba416ea4d06e1ea267309bc2e6f961c57069e2525d17e78748254729d7fdec56720aa85fe6d89b2756a7eeed0a7aa5d13cfb874e3c65576ec8a15d6df17d7d4856653696b10fb9ec205f5e4d1c7a1f3e2dd2994b12eeed93e84776d8dcd8a5d78aecd4f96ae95c0b090d104adf2aa84f0a1fbd8d319fea5476d1a306b2800716e60b00115a5cca678617361c5a89660b4536c56254bc8dd7035d96f05de62b042d16acaeff57c111fdf243b859984063e3fcfdf40c4c4a52889706857a7c3e90e264f30f40cc87bd20e74689f14284bc5ea0a540950dfcc8d33c503477eb1c60", + "mmr_index": 4091742, + "output_type": "Coinbase", + "proof": "7adae7bcecf735c70eaa21e8fdce1d3c83d7b593f082fc29e16ff2c64ee5aaa15b682e5583257cf351de457dda8f877f4d8c1492af3aaf25cf5f496fce7ca54a0ef78cc61c4252c490386f3c69132960e9edc811add6415a6026d53d604414a5f4dd330a63fcbb005ba908a45b2fb1950a9529f793405832e57c89a36d3920715bc2d43db16a718ecd19aeb23428b5d3eeb89d73c28272a7f2b39b8923e777d8eb2c5ce9872353ba026dc79fdb093a6538868b4d184215afc29a9f90548f9c32aa663f9197fea1cadbb28d40d35ed79947b4b2b722e30e877a15aa2ecf95896faad173af2e2795b36ce342dfdacf13a2f4f273ab9927371f52913367d1d58246a0c35c8f0d2330fcddb9eec34c277b1cfdaf7639eec2095930b2adef17e0eb94f32e071bf1c607d2ef1757d66647477335188e5afc058c07fe0440a67804fbdd5d35d850391ead3e9c8a3136ae1c42a33d5b01fb2c6ec84a465df3f74358cbc28542036ae4ef3e63046fbd2bce6b12f829ed193fb51ea87790e88f1ea686d943c46714b076fb8c6be7c577bca5b2792e63d5f7b8f6018730b6f9ddaf5758a5fa6a3859d68b317ad4383719211e78f2ca832fd34c6a222a8488e40519179209ad1979f3095b7b7ba7f57e81c371989a4ace465149b0fe576d89473bc596c54cee663fbf78196e7eb31e4d56604c5226e9242a68bda95e1b45473c52f63fe865901839e82079a9935e25fe8d44e339484ba0a62d20857c6b3f15ab5c56b59c7523b63f86fa8977e3f4c35dc8b1c446c48a28947f9d9bd9992763404bcba95f94b45d643f07bb7c352bfad30809c741938b103a44218696206ca1e18f0b10b222d8685cc1ed89d5fdb0c7258b66486e35c0fd560a678864fd64c642b2b689a0c46d1be6b402265b7808cd61a95c2b4a4df280e3f0ec090197fb039d32538d05d3f0a082f5", + "proof_hash": "cfd97db403c274220bb0dbaf3ecc88e483c0b707d8e6f16dfda37cd4f2c3211c", + "spent": false + } + ] + } + } +} +``` + +Return ### Peer endpoints From bad219cc3e58164396fe971e3704e93117741a72 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 3 Oct 2019 12:14:20 -0400 Subject: [PATCH 07/61] Add get_header --- text/0000-node-api-v2.md | 82 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 81 insertions(+), 1 deletion(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 63fd489..5dae41d 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -139,9 +139,89 @@ Returns the node version and block header version (used by grin-wallet). ### Chain endpoints +#### get_header + +Gets block header given either a height, a hash or an unspent output commitment. +Only one parameters is needed. If multiple parameters are provided only the first one in the list is used. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_header", + "params": [null, "00000100c54dcb7a9cbb03aaf55da511aca2c98b801ffd45046b3991e4f697f9", null], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "cuckoo_solution": [ + 9886309, + 35936712, + 43170402, + 48069549, + 70022151, + 97464262, + 107044653, + 108342481, + 118947913, + 130828808, + 144192311, + 149269998, + 179888206, + 180736988, + 207416734, + 227431174, + 238941623, + 245603454, + 261819503, + 280895459, + 284655965, + 293675096, + 297070583, + 299129598, + 302141405, + 313482158, + 321703003, + 351704938, + 376529742, + 381955038, + 383597880, + 408364901, + 423241240, + 436882285, + 442043438, + 446377997, + 470779425, + 473427731, + 477149621, + 483204863, + 496335498, + 534567776 + ], + "edge_bits": 29, + "hash": "00000100c54dcb7a9cbb03aaf55da511aca2c98b801ffd45046b3991e4f697f9", + "height": 374336, + "kernel_root": "d294e6017b9905b288dc62f6f725c864665391c41da20a18a371e3492c448b88", + "nonce": 4715085839955132421, + "output_root": "12464313f7cd758a7761f65b2837e9b9af62ad4060c97180555bfc7e7e5808fa", + "prev_root": "e22090fefaece85df1441e62179af097458e2bdcf600f8629b977470db1b6db1", + "previous": "0000015957d92c9e04c6f3aec8c5b9976f3d25f52ff459c630a01a643af4a88c", + "range_proof_root": "4fd9a9189e0965aa9cdeb9cf7873ecd9e6586eac1dd9ca3915bc50824a253b02", + "secondary_scaling": 561, + "timestamp": "2019-10-03T16:08:11+00:00", + "total_difficulty": 1133587428693359, + "total_kernel_offset": "0320b6f8a4a4180ed79ecd67c8059c1d7bd74afe144d225395857386e5822314", + "version": 2 + } + } +} +``` + #### get_block -Gets block details given either the height, the hash or an unspent output commitment. +Gets block details given either a height, a hash or an unspent output commitment. Only one parameters is needed. If multiple parameters are provided only the first one in the list is used. ```JSON From bba6a850ec7ba84fdf2a880c0bf006cdf0e22545 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 3 Oct 2019 13:10:57 -0400 Subject: [PATCH 08/61] validate_chain, compact_chain --- text/0000-node-api-v2.md | 66 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 5dae41d..08a9540 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -323,7 +323,71 @@ Only one parameters is needed. If multiple parameters are provided only the firs } ``` -Return +#### get_tip + +Returns details about the state of the current fork tip. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_tip", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "height": 374350, + "last_block_pushed": "000000543c69a0306b5463b92939643442a44a6d9be5bef72bea9fc1d718d310", + "prev_block_to_last": "000001237c6bac162f1add2b122fab6a254b9fcc2c4b4c8c632a8c39855521f1", + "total_difficulty": 1133621604919005 + } + } +} +``` + +#### validate_chain + +Trigger a validation of the chain state. + +```JSON +{ + "jsonrpc": "2.0", + "method": "validate_chain", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` + +#### compact_chain + +Trigger a compaction of the chain state to regain storage space. + +```JSON +{ + "jsonrpc": "2.0", + "method": "compact_chain", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` + ### Peer endpoints From ca69e9e437fb63848dd12071871c219c304239e3 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 3 Oct 2019 14:40:45 -0400 Subject: [PATCH 09/61] Ban/Unban --- text/0000-node-api-v2.md | 41 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 08a9540..037a068 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -388,10 +388,47 @@ Trigger a compaction of the chain state to regain storage space. } ``` - ### Peer endpoints -TBD +#### ban_peer + +Bans a specific peer. + +```JSON +{ + "jsonrpc": "2.0", + "method": "ban_peer", + "params": ["70.50.33.130:3414"], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` + +#### unban_peer + +Unbans a specific peer. + +```JSON +{ + "jsonrpc": "2.0", + "method": "unban_peer", + "params": ["70.50.33.130:3414"], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` ### Pool endpoints From f6184915c87214a303bc9a58e60a8b9a2fdf9931 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 3 Oct 2019 15:43:43 -0400 Subject: [PATCH 10/61] peer endpoint done --- text/0000-node-api-v2.md | 141 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 037a068..affa012 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -390,6 +390,147 @@ Trigger a compaction of the chain state to regain storage space. ### Peer endpoints +### get_peers + +Retrieves information about peers. If `null` is provided, `get_peers` will list all stored peers. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_peer", + "params": ["70.50.33.130:3414"], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": [ + { + "addr": "70.50.33.130:3414", + "ban_reason": "None", + "capabilities": { + "bits": 15 + }, + "flags": "Defunct", + "last_banned": 0, + "last_connected": 1570129317, + "user_agent": "MW/Grin 2.0.0" + } + ] + } +} +``` + +### get_connected_peers + +Retrieves a list of all connected peers. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_peer", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": [ + { + "addr": "35.176.195.242:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "47.97.198.21:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "148.251.16.13:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "68.195.18.155:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "52.53.221.15:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 0, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "109.74.202.16:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "121.43.183.180:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "35.157.247.209:23414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + } + ] + } +} +``` + #### ban_peer Bans a specific peer. From 884a69ea9012127a214065a8df87f2890e90e2cc Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Fri, 4 Oct 2019 19:00:52 -0400 Subject: [PATCH 11/61] Add get_outputs --- text/0000-node-api-v2.md | 89 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index affa012..9424f15 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -348,6 +348,95 @@ Returns details about the state of the current fork tip. } ``` +#### get_kernel + +Returns a `LocatedTxKernel` based on the kernel excess. The `min_height` and `max_height` parameters are both optional. If not supplied, `min_height` will be set to 0 and `max_height` will be set to the head of the chain. The method will start at the block height `max_height` and traverse the kernel MMR backwards, until either the kernel is found or min_height is reached. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_kernel", + "params": ["09c868a2fed619580f296e91d2819b6b3ae61ab734bf3d9c3eafa6d9700f00361b", null, null], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "height": 374557, + "mmr_index": 2211662, + "tx_kernel": { + "excess": "09c868a2fed619580f296e91d2819b6b3ae61ab734bf3d9c3eafa6d9700f00361b", + "excess_sig": "1720ec1b94aa5d6ba4d567f7446314f9a6d064eea69c5675cc5659f65f290d80b0e9e3a48d818cadba0a4e894bbc6eb6754b56f53813e2ee0b1447969894ca4a", + "features": "Coinbase" + } + } + } +} +``` + +#### get_outputs + +Retrieves details about specifics outputs. Supports retrieval of multiple outputs in a single request. Support retrieval by both commitment string and block height. Last field are for whether or not the response will include rangeproof and merkle proof. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_outputs", + "params": [["09bab2bdba2e6aed690b5eda11accc13c06723ca5965bb460c5f2383655989af3f","08ecd94ae293863286e99d37f4685f07369bc084ba74d5c59c7f15359a75c84c03"],376150, 376154, true, true], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": [ + { + "block_height": 374568, + "commit": "09bab2bdba2e6aed690b5eda11accc13c06723ca5965bb460c5f2383655989af3f", + "merkle_proof": null, + "mmr_index": 4093403, + "output_type": "Transaction", + "proof": "e30aa961d6f89361a9a3c60f73e3551f50a3887212e524b5874ac50c1759bb95bc8e588d82dd51d84c7cbaa9abe79e0b8fe902bcfda17276c24d269fbf636aa2016c65a760a02e18338a33e83dec8e51fbfd953ee5b765d97ce39ba0850790d2104812a1d15d5eaa174de548144d3a7d413906d85e22f89065ef727910ee4c573494520c43e36e83dacee8096666aa4033b5e8322e72930c3f8476bb7be9aef0838a2ad6c28f4f5212708bf3e5954fc3971d66b7835383b96406fa65415b64ecd53a747f41d785c3e3615c18dfdbe39a0920fefcf6bc55fe65b4b215b1ad98c80fdafbef6f21ab60596f2d9a3e7bc45d750e807d5eb883dadde1625d4f20af9f1315b8bea08c97fad922afe2000c84c9eb5f96b2a24da7a637f95c1102ecfc1257e19bc4120082f5ee76448c90abd55108256f8341e0f4009cfc3906a598de465467ee1ee072bfd3384e1a0b9039192d1edc33092d7b09d1164c4fc4c378227a391600a8a5d5ba5fe36a2a4eabe0dbae270aefa5a5f2df810cda79211805206ad93ae08689e2675aad025db3499d43f1effc110dfb2f540ccd6eb972c02f98e8151535c099381c8aeb1ea8aad2cfdf952e6ab9d26e74a5611d943d02315e212eb06ce2cd20b4675e6f245e5302cdb8b31d46bb2e718b50ecfad2d440323826570447c2498376c8bad6e4ee97bde41c47f6a20eea406d758c53fb9e8542f114c1a277a6335ad97fdc542c6bbec756dc4a9085c319fe6f0c9e1bb043f01a43c12aa6f4dff8b1220e7f16bc56dee9ccb59fb7c3b7aa6bb33b41c33d8e4b03b6b9cb89491504210dd691b46ffe2862387339d2b62a9dc4c20d629e23eb8b06490c4999433c1b4626fb4d21517072bd8e82511c115ee47bf9a5e40f0a74177f5b573db2e277459877a01b172e026cbb3f76aaf0c61f244584f3a76804dea62175a80d777238", + "proof_hash": "660d706330fc36f611c50d90cb965fddf750cc91f8891a58b5e39b83a5fc6b46", + "spent": false + }, + { + "block_height": 376151, + "commit": "08ecd94ae293863286e99d37f4685f07369bc084ba74d5c59c7f15359a75c84c03", + "merkle_proof": null, + "mmr_index": 4107711, + "output_type": "Coinbase", + "proof": "7083884b5f64e4e61fb910d2c3c603f7c94490716d95e7144b4c927d0ca6ccc0e069cc285e25f38ee90c402ef26005cad2b4073eeba17f0ae3ea2b87095106ef00634f321d8a49c2feaad485bc9ee552564a6a883c99886d0d3a85af3490d718f5a5cbc70f9dcc9bf5d987fb6072132a4c247d4bbd4af927532a887b1e4250b7277771f6b82f43f4fb5a48089ed58e7d3190a19197e07acfed650f8b2cd5f103e994fb3d3735c5727f06f302bd1f182586297dd57a7951ff296bdf6106704abedc39db77f1293effaa7496a77d19420a6208bc1c589b33dad9540cb6180cccf5e085006b01309419f931e54531d770e5fe00eca584072692a7e4883fd65ed4a7c460665608ab96bf0c7d564fe96a341f14066db413a6fddc359eb11f6f962aca70ca1414c35d7941ce06b77d0a0606081b78d5e64a4501f8e8eba9f0e0889042bc54b4cbfd71087a95af63e0306dba214084d4860b0ce66dc80af44224e5a6fef55800650b05cf1639f81bfdc30950f3634d1fd4375d50c22c7f13f3dfb690e5f155a535aff041b7f800bfe74c60f606e8ab47df60754a0e08221c2a50abe643bb086433afd040a7e6290d1d00b3fe657be3bb05c67f90eb183c2acb53c81e1ca15cd8d35fe9d7d52d8f455398e905bdc77ffb211697d477af25704cf9896e8ce797f4fed03e2ba1615e3ad5646eecaa698470f99437d01d5193f041201502763e8bde51e6dc830b5c676d05c8f7f87c4972c578b8d9d5922ba29f6e4a89a123311d02b5ac44a7d5307f7ed5e4e66aaf749afc76c6fc1114445d6fafeea816a0f985eeacdbe9e6d32a8514ca4aaf7faad4e9d43cde55327ac84bac4d70a9319840e136e713aa31d639e43302f3c71a79f08f4e5c9a19a48d4b46403734cd8f3cc9b67bc26ea8e2a01e63a6f5be6e044e8ed5db5f26d15d25de75f672a79315c5e2407e", + "proof_hash": "7cf77fdaecef6c6fc01edca744c1521581f854a9bac0153971edbb1618fc36ad", + "spent": false + }, + { + "block_height": 376154, + "commit": "095c12db5e57e4a1ead0870219bda4ebfb1419f6ab1501386b9dd8dc9811a8c5ff", + "merkle_proof": "00000000003eadc6000000000000000e13c509a17cbb0d81634215cd2482ab6d9eb58b332fcbe6b2c4fa458a63d3cb0dfe3614ebe6e52657870df225d132179fa1ea0fdc2105f0e51d03bc3765a9cd059c60d434a7cae0a3d669b37588c25410f57405c841312cfa50cf514678877a3f4ce8bd3e57723ba75a2b7d61027b2088fbabebdb7336b97ea88b00a7e809a6245def980eba18d987601f4cbd6c3cc9f12a5684fe7a1bc2565a9f8ab63c2db1afa8304f5e23d4754cd97f29c8b06dcb3de4f6d3a83079676b6e9941afe5553a7195384b564ecd6d37522cb5e452cc930d2b549af22698a8fd9bf6cad05a06b09e3f6e672b94e82c0255394b5c187ab76fda653a2491378997ba3d49f9d9c34ca93bc627fe5d98b327c03d429b5473f62672e9d73c4eafd9cb8f62e5158a1ec7eb56653696b10fb9ec205f5e4d1c7a1f3e2dd2994b12eeed93e84776d8dcd8a5d78aecd4f96ae95c0b090d104adf2aa84f0a1fbd8d319fea5476d1a306b2800716e60b00115a5cca678617361c5a89660b4536c56254bc8dd7035d96f05de62b042d16acaeff57c111fdf243b859984063e3fcfdf40c4c4a52889706857a7c3e90e264f30f40cc87bd20e74689f14284bc5ea0a540950dfcc8d33c503477eb1c60", + "mmr_index": 4107717, + "output_type": "Coinbase", + "proof": "073593bc475478f1e4b648ab261df3b0a6e5a58a617176dd0c8f5e0e1d58b012b40eb9b341d16ee22baf3645ea37705895e731dee5c220b58b0f780d781806a10dfa33e870d0494fba18aaa8a7a709bfb3ddf9eb3e4e75a525b382df68dc6f710275cdffb623373c47c1310ae63479826f435ca4520fdc13bb0d995b7d9a10a7587d61bd4a51c9e32c87f3eb6b0f862cdff19a9ac6cb04d6f7fafb8e94508a851dcf5dc6acea4271bb40117a45319da5522b966091b089698f4f940842458b5b49e212d846be35e0c2b98a00ac3d0b7ceaf081272dbed8abd84fe8f26d57bac1340e8184602436ed8c4470ef9dc214df3405de0e71703abec4456b15e122a94706852bb476213ceadf00529d00d8d3b16dc57f4e4a9a86dacfa719e00366728de42f3f830e73f6113f1e391fab07eba1b40f6466203b0ce14701230e934f6138c575660a03dbb0e59d7295df3115a4fc0909a5520d74657b319fc83481079ad6c13400175e39fa2b86071ba563ce8836320713ef8f55d4e90bee3f57df96c7aef0f2e896f57192fae9675471cd9751bcaf2b15e5a65a9733a6f7f9b8147b8f6e8dac51d056018d411fd252225cf88e56f143143f49e8a0d2e43c10de0442dbc84966817532b1256b6769db987526790a389c371a1fe7a36eacffef82877b4db7a9b5e58722ffbd0fc4fdbd7624365ee326bb8b1e60b999f513715b30f37ef6116eabf53b3524b46c33a1fac49205b39e24aa388d823269c1fc43c3599a06b69433a0a47a03bd871321afb7846a6dbfd5891bd84f89c556231745c929d08445f66f332857bfda1c4f86ae58a01007b7303f870ac24e0ba72d84c0ef4903ac2ff777e2c2dcb4d8e303c74e0c8a559686b4d4c25024ee97601787d4e5a97224af41e5d35d91744292f5a41f64d4e1cae77bebebd77a473f3b54e86f7221aac230942f0468", + "proof_hash": "5dd69c083e2c0fd797a499bbafedee0728849afa3476034280ecadf6eb4bffc2", + "spent": false + }, + { + "block_height": 376153, + "commit": "0948cb346b7affe004a6f84fa4b5b44995830f1c332b03537df4c258d51d1afb50", + "merkle_proof": "00000000003eadc4000000000000000dfe3614ebe6e52657870df225d132179fa1ea0fdc2105f0e51d03bc3765a9cd059c60d434a7cae0a3d669b37588c25410f57405c841312cfa50cf514678877a3f4ce8bd3e57723ba75a2b7d61027b2088fbabebdb7336b97ea88b00a7e809a6245def980eba18d987601f4cbd6c3cc9f12a5684fe7a1bc2565a9f8ab63c2db1afa8304f5e23d4754cd97f29c8b06dcb3de4f6d3a83079676b6e9941afe5553a7195384b564ecd6d37522cb5e452cc930d2b549af22698a8fd9bf6cad05a06b09e3f6e672b94e82c0255394b5c187ab76fda653a2491378997ba3d49f9d9c34ca93bc627fe5d98b327c03d429b5473f62672e9d73c4eafd9cb8f62e5158a1ec7eb56653696b10fb9ec205f5e4d1c7a1f3e2dd2994b12eeed93e84776d8dcd8a5d78aecd4f96ae95c0b090d104adf2aa84f0a1fbd8d319fea5476d1a306b2800716e60b00115a5cca678617361c5a89660b4536c56254bc8dd7035d96f05de62b042d16acaeff57c111fdf243b859984063e3fcfdf40c4c4a52889706857a7c3e90e264f30f40cc87bd20e74689f14284bc5ea0a540950dfcc8d33c503477eb1c60", + "mmr_index": 4107716, + "output_type": "Coinbase", + "proof": "72950da23ad7f0d0381e2f788bf0ac6b6bcb17aaccf0373534122a95714d2d0dbf6a24822b4aab0711a595c80bc36122957111c39292f2a36a973252fb88cbda0b1d61ea8ea84f5171a61f751cac97332637b7cf74cc73144b912ba700dedaa60895f06e947f1e42a8c79d70f924f45fdcb6df5d30289f36ff77d0ae368df5775a739b7a25cbfb63f0cdbdc167b046067c2a021fe0950c7b67515b185b9e4a00ce63b795d49ae184fe5cc726d72fc05d717c4fb55dd5f65967dc282d3c47cb6f8a92cb696e5a1d8cca21214bc766e3de6271791cebf646cda97ae77035da16606f3397f71e103137358c97b9943c3e15403184f61230bd0e3954c7681a0891aa7a0cc32e82d830fb7d8759a04d1da7058630a853508df095142f22158c28bd5e3f2477ad6c8990e63d0377a0fa3d588b6584453778eb38cbaec8a33c1d3772c97a826d4a2f6953c35342993b04567e9fea6fc64fb714653f934faa1a8f635d39eb2903de4bed960a3df07dce7c2e3ff517bbc15f467d0190a579bc07b0f1a910b23269d794835bbb34e8318dcc4fd4159f8f03faa77842d445cf61af9e33caf46aa5fae0812a6476a09c0757e929271a96a245701ab14c1fdd836b92b7e763afa623017f68f1bc4eb716ce735820a1311b743dd8d5c6bb275a2e4e7d2eff8f45417b60cc937086c3e7fd3b612ae064d7237eb6a7bd1a39d8575fac312068fa060bc1ceac4df0754601edaf04ecb1b89c0661ea01a593c3763e456bebbd8487edc0ff3bc6f203965cd92b1706070c59a3795f9dee23087cea0aaec015f1b7bfe4df81818d7a37af781ca7b757ace2fa489f85215ecb85976b1c74c7f1df6d834a8bc63e887407ef6e233c55ea040bc5f2471e99ebc92f2283ff592ff751d9226bd105e68e187c91ecb236c9fa4fb060ae4d706c571ac2123da1debd12737d98be118578", + "proof_hash": "0ce421970d13fe9b3981e308c5d0b549982cdda9f69918289cd95ffcd09e0fc2", + "spent": false + } + ] + } +} +``` + #### validate_chain Trigger a validation of the chain state. From 8ed2f5c506d9af21175c838277e1b9b043d46b26 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Mon, 7 Oct 2019 12:38:00 -0400 Subject: [PATCH 12/61] Done with the methods --- text/0000-node-api-v2.md | 206 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 202 insertions(+), 4 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 9424f15..268d9a8 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -405,7 +405,7 @@ Retrieves details about specifics outputs. Supports retrieval of multiple output { "block_height": 376151, "commit": "08ecd94ae293863286e99d37f4685f07369bc084ba74d5c59c7f15359a75c84c03", - "merkle_proof": null, + "merkle_proof": "6b2abbd334c9d75409461fba9c1acd4a8d7bc2ab0bc43143f42388b2a3a87b881505ccf8ffc8737fa6fd4fe412a082d974911bd223eae612d0d1d7ddcc09b5e6079c40b011405b2ccb49ce32473c93aea6d843488d5765fea114d3368d34cd05fcb8c2de3903fbaf39b1f064c809f9f1c0d47959d81a508957040eda55c6dce6dd8c43a79c72faffacfabe1d73055790b6249de2f7c603f186cb109eee58fb1426ea48cb781f88df9acd8996d235fe6bfe60e02aae6e3bfe38ed2599baca1430b3b637072d9bdcdc7644f873728e3cd38eff7124ea848cfad67f8e114cf8595c89a3686a4271cfb2b5098597c315c01d04270ca8f70262af967a947f49adacfa4aad8b6fd196dd0ef4e5cefa132c38c7e5f43db12b3d74f0a8d83c3404e73c6b25a12bff70a8ef4526c89b1558810bb744ede53f8c4cc8cc2555e953637722adb41ea5752281cf1f75599f7e59b17f11f5f9ce4f6b2da4141a3398f51d8b834cdc8b00f61915a41d200572a10bb2102cbae7e94aa7ced3c388dcd58282932c99a8fa66f6fc511ff3e8c60d442bbdb49cca1166328ca8c9bbc97d024570b4cc1ca6c7dba3db223e9e27fd9345b94d3cf10e2b54915b87c57e32965bc2db1b1f956d1962812738ca9b2c93fd7825adf4dffddc97aa85ca0f3f412f02d30678a816d2efbfb6778305fd5e610b6e8af30030bc059880c337bfde326b392d5dcd7c36cb0076fbccc7099b94f1f03bdb525d6e3818b6d50b93ced802957a4b03892c71b6679052bd35e92ceea71a96b22b2ed2c129755f0c74fa172f43da2790f3132a7e57e408d2fc5f1126b088cd0398e6dedcb237242e6720e12e8d7a5a1e196eda6241cfee1cc85e9d20af67f3f9bdf91160516ebcd0b8da6bb7b12229e1112b22c9f1aaef1d75441465cfee2ac1c47b5255514316ed4637e192b00ff28491168f2f2b00", "mmr_index": 4107711, "output_type": "Coinbase", "proof": "7083884b5f64e4e61fb910d2c3c603f7c94490716d95e7144b4c927d0ca6ccc0e069cc285e25f38ee90c402ef26005cad2b4073eeba17f0ae3ea2b87095106ef00634f321d8a49c2feaad485bc9ee552564a6a883c99886d0d3a85af3490d718f5a5cbc70f9dcc9bf5d987fb6072132a4c247d4bbd4af927532a887b1e4250b7277771f6b82f43f4fb5a48089ed58e7d3190a19197e07acfed650f8b2cd5f103e994fb3d3735c5727f06f302bd1f182586297dd57a7951ff296bdf6106704abedc39db77f1293effaa7496a77d19420a6208bc1c589b33dad9540cb6180cccf5e085006b01309419f931e54531d770e5fe00eca584072692a7e4883fd65ed4a7c460665608ab96bf0c7d564fe96a341f14066db413a6fddc359eb11f6f962aca70ca1414c35d7941ce06b77d0a0606081b78d5e64a4501f8e8eba9f0e0889042bc54b4cbfd71087a95af63e0306dba214084d4860b0ce66dc80af44224e5a6fef55800650b05cf1639f81bfdc30950f3634d1fd4375d50c22c7f13f3dfb690e5f155a535aff041b7f800bfe74c60f606e8ab47df60754a0e08221c2a50abe643bb086433afd040a7e6290d1d00b3fe657be3bb05c67f90eb183c2acb53c81e1ca15cd8d35fe9d7d52d8f455398e905bdc77ffb211697d477af25704cf9896e8ce797f4fed03e2ba1615e3ad5646eecaa698470f99437d01d5193f041201502763e8bde51e6dc830b5c676d05c8f7f87c4972c578b8d9d5922ba29f6e4a89a123311d02b5ac44a7d5307f7ed5e4e66aaf749afc76c6fc1114445d6fafeea816a0f985eeacdbe9e6d32a8514ca4aaf7faad4e9d43cde55327ac84bac4d70a9319840e136e713aa31d639e43302f3c71a79f08f4e5c9a19a48d4b46403734cd8f3cc9b67bc26ea8e2a01e63a6f5be6e044e8ed5db5f26d15d25de75f672a79315c5e2407e", @@ -437,6 +437,52 @@ Retrieves details about specifics outputs. Supports retrieval of multiple output } ``` +#### get_unspent_outputs + +UTXO traversal. Retrieves last utxos since a start index until a max. Last boolean is optional to whether or not return the rangeproof. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_unspent_outputs", + "params": [1, 2, true], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "highest_index": 2078061, + "last_retrieved_index": 30, + "outputs": [ + { + "block_height": 1, + "commit": "08b7e57c448db5ef25aa119dde2312c64d7ff1b890c416c6dda5ec73cbfed2edea", + "merkle_proof": null, + "mmr_index": 1, + "output_type": "Coinbase", + "proof": "9330ad8cde205f317c6537eca96b866293a0489615a9a277b4d3a597c873544c82474932b641e06ac8719604ee52e895e8cd4621b6bfb85780cd9becce14d0700b83a664db2f52a26c425fd777ad88944cdfff38043a2793ed4d9aa67e36cbfd5585579fc69dda930418af5eaf603654f6f751258d2dfc8c2113c171e130f31ec1e6cce2a718e435298fce5d64ffe1bd3464fd7c87cfa92093855be034bfe4439e928bd92ad77fd0a0e00355ee1d1a9ceb1ed0c408dcfdba8c583e7598dc700aaa9f91432097259a405f5b7315a2f7658861e3349bb0dc8bf883726a215f0149ded6613e5ac0670c0c5202247d7c27c8a7d03bdb03c9cf5455463f9b42cf87403e31f8383cc4f49a34c62ae459f5801a9eed4f0ee3dfd5f55b7011c0cae393c474abd6f8c7965b9b5fff3104dd4e39542077c0c8dd2f8ffceb6bb598512d90506d0a7184f20f1498cf458787f23284b54888c9be416d103f760406357a16b6d841a303d5c95b6b474d2d7f0fea0a2a76c897dd2110e9303f54684169421147684c6f1819c33cef3f38ec995a508450c02cd1872f8065fdee723109c18b1dd2ddde75825546ecf0df0793c353b20c946cd64122cea8c116f432336899a16ad24a2aafcb8f900e09a1147135fcf2a54cbf81db308a47a08a49c77c130e5dc5e661cd55a5cc69e607055a5b08111bf61a62ea5778f85119043633f1cab8c756d756c5a34851024ac311a596b1cd919bbca43226f0ba057f6b57de2f6955b0823c3826de7f6096c1c1b6b9b8e4063e1645c0bff32f80561aaa959d97120fbc2ecd9d2be28bd0c17811dc59a88049f6d8952ee9a0a0207693c89ca3ad1197e9bfdfc03be9d845aea8d663969217e3b494cee9e652bc9f8713e2fd5cb1843848f46c3a6ab024d0e3d57ca45454cdbda414adaa835fa147deb4ffb7129cf3a8d86726a0144794", + "proof_hash": "6c301688d9186c3a99444f827bdfe3b858fe87fc314737a4dc1155d9884491d2", + "spent": false + }, + { + "block_height": 29, + "commit": "09bab1ddad0f6fec1aedcd3830c5c647515ad543929e722344e4a8d390b6fdd51b", + "merkle_proof": null, + "mmr_index": 55, + "output_type": "Coinbase", + "proof": "4a5f858d4311bdd902f4446682f27f64be376283b1171060fd2ad33d85350fee13c25a030874d6308d2b325995a3fe545eb1d85ba66e2ba002b794edfdeacb3f0fd2a690b9a78137771b3633aaef2a77f62fbe4d6b4b373c4bdb7e5f58cfae361a3b4c2e4420cc0d38465b2444e01b50e57c6ebfc2afd6dda9017e54585638bddef17d181d1fd7064d975d8bb1dcfd96c89486aed4680b4d39294a141581d1f51c1acfbb80e2ffc40f8499cdc43be04cacda1e34dd6592edfc500229aa70db1c2869f974cfe9aee0cab696c198624de8ecdaf5ae481a1e46fe79fe983209459b89492f2b24416c368394c43c60c33d0fdd1792f0a58d11763e7c8b89d27da25109db346e4d7b62935d182b45dfb659829c55922350e6f7e3452d9311e527ec5b561f4d043cef865f683fce1ce2d410d414f5bcee63c4bbc00964b0fa757bdfd68158e22c1068d871a45759fbd527883c0451db6f36b15139864b6177a78ad64d326e0152914e5313a97ed7b685e5089f2758bf072c804560306bd944831f067c3413ded09330fd788f353e4ee875d3c9303dd4ec0dda9d55b4a27d7748b3247fe85cf3d26b7004e6e3379041fad136fccdacd02b06456a50ad40a3259842c0794f2d59dbd8fa6b4af065b38c388d76b82136b633b06779e4eb05b5b62ec37cdc2986327639bafa8651318f4c00c066e6f45504ec9a96874d5510b519f434a1a88175d51f86e8ee36ae18d107cfaf83e60b2e62fff032c7539be66d776e3a52c5f9b0ee6fe08820d65cd75d35c793e5ab3914adf5a97b7dba75e90d4a4c9aa844e2f1e9464cd5fc4923b475defca4e3b03e1b33353ff91ac1084712cf4445e329ffdbe1e2da16ae71dee0e914b546fdc0db9b0fcde80822ee716e9f2eec90db7aa4417d53a1266e1e8383e20c9a9548bae35c2a8e1293a49e7afbd8011a9e66e79ed6be", + "proof_hash": "a64ed774d824dc55123c6c5ba46d84bac15b6ead8cb60200836c2a0e74506ab0", + "spent": false + } + ] + } + } +} + +``` + #### validate_chain Trigger a validation of the chain state. @@ -662,11 +708,163 @@ Unbans a specific peer. ### Pool endpoints -TBD +#### get_pool_size + +Returns the number of transaction in the transaction pool. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_pool_size", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": 1 + } +} +``` + +#### get_stempool_size + +Returns the number of transaction in the stem transaction pool. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_stempool_size", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": 0 + } +} +``` + +#### get_unconfirmed_transactions + +Returns the unconfirmed transactions in the transaction pool. Will not return transactions in the stempool. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_unconfirmed_transactions", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": [ + { + "src": "Broadcast", + "tx": { + "body": { + "inputs": [ + { + "commit": "0992ce1827ec349e9f339ce183ffd01db39bf43999799d8191bfc267a58f0a715c", + "features": "Coinbase" + }, + { + "commit": "0943a3c4ee4a22a5b086c26f8e6dc534204dafde0cf4c07e0c468d224dd79127ec", + "features": "Plain" + } + ], + "kernels": [ + { + "excess": "083c49eaaf6380d44596f52cce4cf278cfac6dd34fbef73981002d8f1e8ee8abe4", + "excess_sig": "3f011e7e288231d67f42cb4f6416c4720e6170d5e3c805a52d33aa4521328f9be0303be654bc8ddcd3111aadc27c848b9cf07e0a70885ef79be70b7bb70f8c75", + "features": { + "Plain": { + "fee": 7000000 + } + } + } + ], + "outputs": [ + { + "commit": "0873fafd4a0e4f365939e24c68eeb18aafc6674ca244a364dcdbfa8fa525e7bae1", + "features": "Plain", + "proof": "4b675be40672d5965c43d9f03880560a8ac784ee3de8768e28c236a4bc43b8c3d4bc83dee00d2b96530af9607c3b91d9a828f0234bf2aaf7e7c0e9cf936db69c04ca1b267668fbdb2f08ce05c8b119c9d886ceaafb4634b7fae7ea01966ad825dddc9ffab8093155d9c5d268160b86fcad95f4f5e66bf46ff642a51629dbdfd7bba7936846915b925d547337a1b95c33030fad4178468825936242e631797aa3a8f0a5ae0d23040938622648c8432fc247a902abad27e383affb4ec518e4f6f55f55e264bc0f99957be203cfb26d4b8e561fb36da55a50b6ef5861134c484556d701133e1dceda5ea53e731184e0a11f33d06e13ca37d03d39dd047170580534b049862fcd6c73decc7c0af45a267ed148fe6ef2cc375ffebfa8187d2fa0a134428a036d2ec1f65d3ce036b955730fc1ee43b23b574bae2b58b7adfa2a7a45cdec393d9b658857c911560aa3c44cf4435a99d68f3dbc81c82ea43e426ef0198148a90336ee72472aab5f7feea1df93ec830fe5ec642c93c1046dec955df361bfdc3ab74477f847a1b72e8735ef65a8a6d1680745c0152bfb5cbb2a4b4671491a253a1a09d5a07d55f4872c9f0a3d25e07b257926629d5bb96aed96f5debab02503eb0ac45033323cc5a46c8e5d4469ee9f3dd618a20d54d6f5740c010fe5a0fe853efeb253a6df196bd24469ac51c1be8ba84737cecdb5ab73d7c52570d2273621fb69bd7ed985bbc6999dbd2d6fd2687ae44a391d604ff232cc6b3fbedd5d1cd0cd8c658c5d56069b5a5099cc5c9f48bbf7d7e83b4f9a7bdef6eabd164c8395468f818e8cd8c1c800bc3adfd66dbcb247d1bda5a7af38c288c0beb8e0d9160bf67500094530a0f8be52e97b5c2114f5a4a333a11c7f37f4c47a437422455d8cbcfa770cdc85ec55accf48cf14550b07f1346a02fccdf280fcb24c1fb38751d889a17e" + }, + { + "commit": "08de9e42d361cabd99e566c67f7f8599c7e6985cd285a841277f1aeb89ad6c8fe3", + "features": "Plain", + "proof": "5eb7afa00e9681e3b6425fb4256c96905303505787d6a065e88a50154410b9a371b0f879d3f97cfa00425e9c8266e180188656acdbb46cacfdfb159fb135c5eb03b08be3c231c4b21df777da2e2afe8d30db91e602dc4ceed71aeb1b45a0266cfeadc4acbf9fdf7a67f67408fbbea7bf14182bc407373d243c6875373b655695604deb575369a9b28274885601b338882219c7f508aa2a0ae1d02736af2249327145f1d3d00093f9587f0e0b408692700fac0f2a048c329e81cabaa4b997dd88923fe97420125f394e21b4835e36cce9de383d9e223df1b5a6ba6f48ffeac315991189dc2716cc7ec07f6ccc8062344d5ed4fcaddf9070f44f0c59ffe8160d1f6fdfe42b40066f51e687d38b6b5255771800ac060bd8034cd68d14eee1b2f43b6d7bf20d71549ea9a50006dd30b9a795e785385801546eb9a83721a09fc34d3b69d4ccdc0ff0fb74d224048aeb66ecff5515296cadd57f42e0717cbba7c70719a10c007db4520e868efe98a51001b67952d7bda3174195a3d76b93ee4dac60137a38b2e8309cad13ef1cfb6c467f1969385e5b334b52f4fd55da440e036d2a428e9f3be905d79f717c169060468acc6d469636fed098b1aba5cd055a120314bcab55d5b8b6889321edf373517e93ef67fbe74557ec6c0211265efefa25a34ac267cf1db891c47163bfed20d2b535abfe60390c2844dcef5f0aad5fa7f1db9f726d7f223c025861069603936a22377707cdd3915e762e7061132124c716212b0e91bb7fc5d7816366f5d169d93fe75669a6ba19057bb2450958aa6f5ada09042570f46215af5a41b623d140be574b7a8c9ab24ea48da416dbe6ec0fa3b889206fb804df8d69805ceb80f1e9d4e8b664b3939491cba946d87585c830e3dab0638fa279b5e911642f18452e2731764aa62f92bbcf194c97f344c90c1931fd2c3af4bcf6b0" + } + ] + }, + "offset": "0eb2c2669ce918675c72697891e5527bd13da5a499396381409219b8bbbd8129" + }, + "tx_at": "2019-10-07T16:20:08.709114Z" + } + ] + } +} +``` + +#### push_transaction -### TxHashSet endpoints +Push new transaction to our local transaction pool. Optional fluff boolean to bypass Dandelion relay (false by default). -TBD +```JSON +{ + "jsonrpc": "2.0", + "method": "push_transaction", + "params": [ { + "body": { + "inputs": [ + { + "commit": "0904cbd34d0745eb00ffc3e95c9f4746738794d00268e243e9b57163a73b384102", + "features": "Coinbase" + } + ], + "kernels": [ + { + "excess": "08385257d22f1b8a758903f78ae12545245d620cffc50e7ee7bc852c5815513dc7", + "excess_sig": "e001a7349fd40d4a9dfc1df275d30906fb3b304f8c7892a20ed5c9b10923c871cbabedcf322511a9ce56f10113b48855441f681280133e121b25ea1ff7efad9e", + "features": { + "Plain": { + "fee": 8000000 + } + } + } + ], + "outputs": [ + { + "commit": "087c3ca7419751e96cdae4908bb8a92fc2826f2ad36690420b905d51beb7409ca0", + "features": "Plain", + "proof": "379ae236937883c2e1e613fb30f1b18d2a44d4173360e94bcd07862aafaf81b3aaa1154d67287cc03efde0d3981c6da8a18e2e426f5c30afc0f2e3a75012448402d8d56df52b87f4815575a56d4da174f8187e4faae64bf883b249ceed694271f84ef62a3711d36c997dff7a11111419011e36e3a070b7552415a55faaa3999f99439edccdfe5313277147fdb42be1798442bb225c2b546f5347920584b365aa81a0365b4a706c97c89617b0e6218d2c9bc15805caab27c438ed06340cc4f8dc7bfca0e9d38864c88bb0c834372f6b662b9159134f3f8ec9b8a87878739a7e516b97419ac29e1d4a2b250321470a9a6b98d07065bb7e79afc25a5ab6fc47108f53223078a64502bd4af1a109641447dab82741ebe3fbdbd803ee7a42fe2554e78fa86bd1d1e6e3b913118e9419b0be6f976b2404447d943b5f1bac19a5809fd6834797945a62d21b1ecb6ddebbc5ef94ca9e704d033bd64afde67bd3e06e2cca3bb10190188afc0af80b48dd862b86753d8b4af314763324deb1c97cf020cb87285a47cd28874bb91c6cdf858965e8b9daafbcbc1b4817d334a97d7e25e01b2d072d8dcc6418e3dc7b8e7712632f939238e65ed0731c7af02d55a8884cd8f7f88dc0f63a21955a7364562532f5716c89e14f8f23ad78f6fe2f1649e13ea8f8185f3ee63cc174684d1ef8d8c33fb25bc802f8e05e53fe200b1ea5231f588a020942e6fd7eec67301700088dae8816c16a337120063c21e1604e009df932032812f88be6473af13f802b42d8ad6fc14230fbe13ede178319a7b6540656234ec1f2fcfa70f6faa9c4b6b8150b81fe0fdc273a9bb385d766a02041a5c3f58471d42059c17d84d13ad592aa0ccf337970e7eef06f306b13288795123c9c005b815d848f359b23450656b310f09cda9ad4b7b6931805d47dcd10a8745d834a984e2055168ac3" + }, + { + "commit": "09a7b2c1d4b346c4ebe9c6c979e32e7740446624d5439d9d7abb82166c2545e5be", + "features": "Plain", + "proof": "5fb0ee4093a153e2ed173207dbfa02b4d185f1f313ea4cbf222558819074543f19e9bcdb595a23d4ee971aafcc614b6d2774e22cee6627bc4388297fe6ebf03e0d422f3eb8003cc8516417a6b32eb22f87e1745e0ae5bf1733f2ea253399719b1ef0067934dc548c58729604d24a44040165b32d05e82c9efc9a1f30151dd73ce893ae94709ec2fe5d0f409bb54a86604f0e92915b4f93e7adde823eccf87830ae91d71a7b99967dbcc8531fee44c20c24fb6fe2a34fe86ba5da3a9235cbcdcde033ead57d65c03903a9c9ed877bf0fab9f26d08552c64ea668d5408c84b74bc3ac8335aaaa04ebcf523d36d2207fb8770e976b6fde7d04e2148de5a4169c60b1958bb840b79a8c8f356e1f1fadc35a5a7e276fcd67c354cde546548c9bf788981f38edf5a406977826aa4524004e770b3d3cd6b26f0dc99729ffd9929fa4509b145ef0c3e4293e71b964da731a47cc9f082350acf32afb64b3b12f8383c8f2cc9880131a80ea957b2908c92f21d2db7aa5d67bafb11eb07674e52b920e67a86259dd9c5dcdd18bad182fd85ec4b659c47ea2e2e8a89c57e4d2cde87958fc2ab932e169f6805d2fb14549ac93807bc426eb4cf6d29ff6a4cf22e35dbb27f04211b06b65173501c17a3bb3ff0eecc9bb05dca23379abe457ca3010ebea69e1a2f7f3ed6531bf766007cdd1ac7d6c762785fb56f36194cc2ccaee76a499a7383288e84981b103d76cbe007f66c913eacb277746e78ae08627b279ac1f9a43ab284d8a3b32c6edcd2ea99e8ea836b31a1e2582be6c41f2282cf5fc7bdb95e4b412a5eeccad29670197873a888a100c4b2704ce75137fc997a5632d81001f9b57300a9bf99edd857065be83f835e4c49d852165ba18e1c96316c153459a913773d5d86ddc26c5cd1fff38a8fbb62506b0aef6076382674c0fa95a50a03b0c3df0a688a2cbf" + } + ] + }, + "offset": "0ec14d3875ad5a366418256fe65bad2a4d4ff1914e1b9488db72dd355138ca3a" + }, + true + ], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` ## Authentication From 9bf791eafdbaa8ab6f647bd8745e6ba0d1600253 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 9 Oct 2019 12:32:06 -0400 Subject: [PATCH 13/61] Fix typos --- text/0000-node-api-v2.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 268d9a8..5547c67 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -532,7 +532,7 @@ Retrieves information about peers. If `null` is provided, `get_peers` will list ```JSON { "jsonrpc": "2.0", - "method": "get_peer", + "method": "get_peers", "params": ["70.50.33.130:3414"], "id": 1 } @@ -564,7 +564,7 @@ Retrieves a list of all connected peers. ```JSON { "jsonrpc": "2.0", - "method": "get_peer", + "method": "get_connected_peers", "params": [], "id": 1 } From 07d176bcce9b1f49d90411a06528419ceaf6ee00 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 9 Oct 2019 14:00:07 -0400 Subject: [PATCH 14/61] Add TOC ready for review --- text/0000-node-api-v2.md | 74 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 5547c67..83c5241 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -7,6 +7,45 @@ --- +Table of Contents + +- [Summary](#summary) +- [Motivation](#motivation) +- [Community-level explanation](#community-level-explanation) +- [Reference-level explanation](#reference-level-explanation) + * [Current Endpoints with the v1 API](#current-endpoints-with-the-v1-api) + * [Proposed Endpoints](#proposed-endpoints) + + [Miscellaneous endpoints](#miscellaneous-endpoints) + - [get_status](#get-status) + - [get_version](#get-version) + + [Chain endpoints](#chain-endpoints) + - [get_header](#get-header) + - [get_block](#get-block) + - [get_tip](#get-tip) + - [get_kernel](#get-kernel) + - [get_outputs](#get-outputs) + - [get_unspent_outputs](#get-unspent-outputs) + - [validate_chain](#validate-chain) + - [compact_chain](#compact-chain) + + [Peer endpoints](#peer-endpoints) + + [get_peers](#get-peers) + + [get_connected_peers](#get-connected-peers) + - [ban_peer](#ban-peer) + - [unban_peer](#unban-peer) + + [Pool endpoints](#pool-endpoints) + - [get_pool_size](#get-pool-size) + - [get_stempool_size](#get-stempool-size) + - [get_unconfirmed_transactions](#get-unconfirmed-transactions) + - [push_transaction](#push-transaction) + * [Authentication](#authentication) + + [Wallet support](#wallet-support) + + [Legacy support](#legacy-support) + + [API only](#api-only) +- [Drawbacks](#drawbacks) +- [Prior art](#prior-art) +- [Future possibilities](#future-possibilities) +- [References](#references) + # Summary [summary]: #summary @@ -77,6 +116,41 @@ This endpoints can be grouped into 5 categories: ## Proposed Endpoints +This RFC proposes to keep the logic of the following endpoints and refactor them: + +```JSON +[ + "get blocks", + "get headers", + "get chain", + "post chain/compact", + "get chain/validate", + "get chain/kernels/xxx?min_height=yyy&max_height=zzz", + "get chain/outputs/byids?id=xxx,yyy,zzz", + "get chain/outputs/byheight?start_height=101&end_height=200", + "get status", + "get txhashset/outputs?start_index=1&max=100", + "get pool", + "post pool/push_tx", + "post peers/a.b.c.d:p/ban", + "post peers/a.b.c.d:p/unban", + "get peers/all", + "get peers/connected", + "get peers/a.b.c.d", + "get version" +] +``` + +The logic of the following endpoints will NOT be implemented as they are purely internals: + +```JSON + "get txhashset/roots", + "get txhashset/lastoutputs?n=10", + "get txhashset/lastrangeproofs", + "get txhashset/lastkernels", + "get txhashset/merkleproof?n=1", +``` + ### Miscellaneous endpoints #### get_status From 867f479732f2d92bca76df3b694de90bd766d3f8 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 9 Oct 2019 14:02:09 -0400 Subject: [PATCH 15/61] Fix TOC links --- text/0000-node-api-v2.md | 52 ++++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 83c5241..3c922a1 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -7,7 +7,7 @@ --- -Table of Contents +# Table of Contents - [Summary](#summary) - [Motivation](#motivation) @@ -16,34 +16,34 @@ Table of Contents * [Current Endpoints with the v1 API](#current-endpoints-with-the-v1-api) * [Proposed Endpoints](#proposed-endpoints) + [Miscellaneous endpoints](#miscellaneous-endpoints) - - [get_status](#get-status) - - [get_version](#get-version) + - [get_status](#get_status) + - [get_version](#get_version) + [Chain endpoints](#chain-endpoints) - - [get_header](#get-header) - - [get_block](#get-block) - - [get_tip](#get-tip) - - [get_kernel](#get-kernel) - - [get_outputs](#get-outputs) - - [get_unspent_outputs](#get-unspent-outputs) - - [validate_chain](#validate-chain) - - [compact_chain](#compact-chain) - + [Peer endpoints](#peer-endpoints) - + [get_peers](#get-peers) - + [get_connected_peers](#get-connected-peers) - - [ban_peer](#ban-peer) - - [unban_peer](#unban-peer) - + [Pool endpoints](#pool-endpoints) - - [get_pool_size](#get-pool-size) - - [get_stempool_size](#get-stempool-size) - - [get_unconfirmed_transactions](#get-unconfirmed-transactions) - - [push_transaction](#push-transaction) + - [get_header](#get_header) + - [get_block](#get_block) + - [get_tip](#get_tip) + - [get_kernel](#get_kernel) + - [get_outputs](#get_outputs) + - [get_unspent_outputs](#get_unspent_outputs) + - [validate_chain](#validate_chain) + - [compact_chain](#compact_chain) + + [Peer endpoints](#peer_endpoints) + + [get_peers](#get_peers) + + [get_connected_peers](#get_connected_peers) + - [ban_peer](#ban_peer) + - [unban_peer](#unban_peer) + + [Pool endpoints](#pool_endpoints) + - [get_pool_size](#get_pool_size) + - [get_stempool_size](#get_stempool_size) + - [get_unconfirmed_transactions](#get_unconfirmed_transactions) + - [push_transaction](#push_transaction) * [Authentication](#authentication) - + [Wallet support](#wallet-support) - + [Legacy support](#legacy-support) - + [API only](#api-only) + + [Wallet support](#wallet_support) + + [Legacy support](#legacy_support) + + [API only](#api_only) - [Drawbacks](#drawbacks) -- [Prior art](#prior-art) -- [Future possibilities](#future-possibilities) +- [Prior art](#prior_art) +- [Future possibilities](#future_possibilities) - [References](#references) # Summary From 343d1a085ee82de60a0dea5be9cd582232a0f459 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 9 Oct 2019 14:03:31 -0400 Subject: [PATCH 16/61] Clarify text --- text/0000-node-api-v2.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 3c922a1..8cf29c4 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -62,7 +62,11 @@ The current Node API (referred here as v1 API) is a REST API. This API while sim - Contains call with heterogenous args such as `?byid=xxx` and `commitment/xxx` which can be confusing and lack some uniformity. - Uses REST which is bound to HTTP while v2 wallet API uses JSON-RPC. -The goal of this RFC is to provide a new API with cleaner methods and errors, automatically generated documentation directly on docs.rs and finally stronger basis for future improvements. +The goal of this RFC is to provide a new API with: +- Cleaner methods and errors. +- Generated documentation directly on docs.rs. +- Automatic testing with doc tests. +- Finally stronger basis for future improvements. # Community-level explanation From 24b1d028bf83018da9918cfceb49c59ab6a9a44b Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 9 Oct 2019 14:08:32 -0400 Subject: [PATCH 17/61] Fix typo and links --- text/0000-node-api-v2.md | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 8cf29c4..e53cd1d 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -27,23 +27,23 @@ - [get_unspent_outputs](#get_unspent_outputs) - [validate_chain](#validate_chain) - [compact_chain](#compact_chain) - + [Peer endpoints](#peer_endpoints) - + [get_peers](#get_peers) - + [get_connected_peers](#get_connected_peers) + + [Peer endpoints](#peer-endpoints) + - [get_peers](#get_peers) + - [get_connected_peers](#get_connected_peers) - [ban_peer](#ban_peer) - [unban_peer](#unban_peer) - + [Pool endpoints](#pool_endpoints) + + [Pool endpoints](#pool-endpoints) - [get_pool_size](#get_pool_size) - [get_stempool_size](#get_stempool_size) - [get_unconfirmed_transactions](#get_unconfirmed_transactions) - [push_transaction](#push_transaction) * [Authentication](#authentication) - + [Wallet support](#wallet_support) - + [Legacy support](#legacy_support) - + [API only](#api_only) + + [Wallet support](#wallet-support) + + [Legacy support](#legacy-support) + + [API only](#api-only) - [Drawbacks](#drawbacks) -- [Prior art](#prior_art) -- [Future possibilities](#future_possibilities) +- [Prior art](#prior-art) +- [Future possibilities](#future-possibilities) - [References](#references) # Summary @@ -66,7 +66,7 @@ The goal of this RFC is to provide a new API with: - Cleaner methods and errors. - Generated documentation directly on docs.rs. - Automatic testing with doc tests. -- Finally stronger basis for future improvements. +- Stronger basis for future improvements. # Community-level explanation @@ -110,7 +110,7 @@ While the endpoints are documented in details [here](https://github.com/mimblewi ] ``` -This endpoints can be grouped into 5 categories: +These endpoints can be grouped into 5 categories: - miscellaneous endpoints (which contain `status` and `version` endpoints) - `chain` endpoints (which also contain `blocks`, `chain` and headers` endpoints) @@ -220,7 +220,7 @@ Returns the node version and block header version (used by grin-wallet). #### get_header Gets block header given either a height, a hash or an unspent output commitment. -Only one parameters is needed. If multiple parameters are provided only the first one in the list is used. +Only one parameter is required. If multiple parameters are provided only the first one in the list is used. ```JSON { @@ -300,7 +300,7 @@ Only one parameters is needed. If multiple parameters are provided only the firs #### get_block Gets block details given either a height, a hash or an unspent output commitment. -Only one parameters is needed. If multiple parameters are provided only the first one in the list is used. +Only one parameter is required. If multiple parameters are provided only the first one in the list is used. ```JSON { @@ -603,7 +603,7 @@ Trigger a compaction of the chain state to regain storage space. ### Peer endpoints -### get_peers +#### get_peers Retrieves information about peers. If `null` is provided, `get_peers` will list all stored peers. @@ -635,7 +635,7 @@ Retrieves information about peers. If `null` is provided, `get_peers` will list } ``` -### get_connected_peers +#### get_connected_peers Retrieves a list of all connected peers. @@ -788,7 +788,7 @@ Unbans a specific peer. #### get_pool_size -Returns the number of transaction in the transaction pool. +Returns the number of transactions in the transaction pool. ```JSON { @@ -808,7 +808,7 @@ Returns the number of transaction in the transaction pool. #### get_stempool_size -Returns the number of transaction in the stem transaction pool. +Returns the number of transactions in the stem transaction pool. ```JSON { @@ -946,7 +946,7 @@ Push new transaction to our local transaction pool. Optional fluff boolean to by ## Authentication -Similarly to the V1 API, the v2 API will use basic auth with the same secret. This token is usually in `grin/main/.api_secret`. +Like the V1 API, the v2 API will use basic auth with the same secret. This token is usually in `grin/main/.api_secret`. ### Wallet support @@ -964,7 +964,7 @@ Note that this RFC does not propose making user-facing changes to the existing C [drawbacks]: #drawbacks -The implementation of this RFC will temporarily introduces some additional code complexity as v1 and v2 node API will need to coexist until a cutoff release is determined. +The implementation of this RFC will temporarily introduce some additional code complexity as v1 and v2 node API will need to coexist until a cutoff release is determined. # Prior art From 7972280d79d95e8e8c4889243b2311f4207aa268 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 9 Oct 2019 17:27:28 -0400 Subject: [PATCH 18/61] Add error examples --- text/0000-node-api-v2.md | 89 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 86 insertions(+), 3 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index e53cd1d..273a7e1 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -37,6 +37,9 @@ - [get_stempool_size](#get_stempool_size) - [get_unconfirmed_transactions](#get_unconfirmed_transactions) - [push_transaction](#push_transaction) + * [Errors](#errors) + + [JSON-RPC Errors](#json-rpc-errors) + + [API Result Errors](#api-result-errors) * [Authentication](#authentication) + [Wallet support](#wallet-support) + [Legacy support](#legacy-support) @@ -80,7 +83,7 @@ This new API will be particularly useful for wallet developer and should ultimat ## Current Endpoints with the v1 API -While the endpoints are documented in details [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md), here is an overview of the current REST API Endpoints of the V1 API. +While the endpoints are documented in details [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md), here is an overview of the current REST API Endpoints of the v1 API. ```JSON [ @@ -944,9 +947,89 @@ Push new transaction to our local transaction pool. Optional fluff boolean to by } ``` +## Errors + +There is two kind of errors that can appear when making a call on the API v2: basic JSON-RPC errors and API result errors + +### JSON-RPC Errors + +These errors are often due to a miscontructed JSON body. +For example when there is not enough parameters: + +```JSON +{ + "error": { + "code": -32602, + "message": "WrongNumberOfArgs. Expected 4. Actual 5" + }, + "id": 1, + "jsonrpc": "2.0" +} +``` + +or invalid type of data entered: + +```JSON +{ + "error": { + "code": -32602, + "message": "InvalidArgStructure \"start_height\" at position 1." + }, + "id": 1, + "jsonrpc": "2.0" +} +``` + +or method not found: + +```JSON +{ + "error": { + "code": -32601, + "message": "Method not found" + }, + "id": 2, + "jsonrpc": "2.0" +} +``` + +this list of errors is not exhaustive, for more information about the possible error objects see the [JSON RPC 2.0 specifications](https://www.jsonrpc.org/specification#error_object). + + +### API Result Errors + +These type of errors are due to an API error during the query. This type of error is wrapped into the result. +The possible error objects are defined in the [`grin_api` crate](https://docs.rs/grin_api/2.0.0/grin_api/struct.Error.html) + +For example, trying to ban a peer that's already banned: + +```JSON +{ + "id": 2, + "jsonrpc": "2.0", + "result": { + "Err": { + "Internal": "ban peer error: PeerNotFound" + } + } +} +``` + +or a block that doesn't exist: + +```JSON +{ + "id": 2, + "jsonrpc": "2.0", + "result": { + "Err": "NotFound" + } +} +``` + ## Authentication -Like the V1 API, the v2 API will use basic auth with the same secret. This token is usually in `grin/main/.api_secret`. +Like the v1 API, the v2 API will use basic auth with the same secret. This token is usually in `grin/main/.api_secret`. ### Wallet support @@ -954,7 +1037,7 @@ The wallet currently uses the Grin Node API v1 so changes will be needed in the ### Legacy support -The V1 API will remain active for a time the mode of operation for its REST API will be assumed to work as currently. This setup should allow existing wallets and apps to continue working as-is until a cutoff release for legacy mode is determined. +The v1 API will remain active for a time the mode of operation for its REST API will be assumed to work as currently. This setup should allow existing wallets and apps to continue working as-is until a cutoff release for legacy mode is determined. ### API only From f0b41d7b2e2d448f2277d515284dde86594bcfcc Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 16 Oct 2019 10:52:58 -0400 Subject: [PATCH 19/61] Comment + formatting --- text/0000-node-api-v2.md | 41 ++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 273a7e1..5541ebf 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -41,9 +41,10 @@ + [JSON-RPC Errors](#json-rpc-errors) + [API Result Errors](#api-result-errors) * [Authentication](#authentication) - + [Wallet support](#wallet-support) - + [Legacy support](#legacy-support) - + [API only](#api-only) + * [Wallet support](#wallet-support) + * [Legacy support](#legacy-support) + * [API only](#api-only) + * [Node V1 Depreciation Timeline](#node-v1-depreciation-timeline) - [Drawbacks](#drawbacks) - [Prior art](#prior-art) - [Future possibilities](#future-possibilities) @@ -158,6 +159,29 @@ The logic of the following endpoints will NOT be implemented as they are purely "get txhashset/merkleproof?n=1", ``` +The new endpoint methods are the following: + +- [get_status](#get_status) +- [get_version](#get_version) +- [get_header](#get_header) +- [get_block](#get_block) +- [get_tip](#get_tip) +- [get_kernel](#get_kernel) +- [get_outputs](#get_outputs) +- [get_unspent_outputs](#get_unspent_outputs) +- [validate_chain](#validate_chain) +- [compact_chain](#compact_chain) +- [get_peers](#get_peers) +- [get_connected_peers](#get_connected_peers) +- [ban_peer](#ban_peer) +- [unban_peer](#unban_peer) +- [get_pool_size](#get_pool_size) +- [get_stempool_size](#get_stempool_size) +- [get_unconfirmed_transactions](#get_unconfirmed_transactions) +- [push_transaction](#push_transaction) + +NB: The following v2 endpoints are classified by categories solely to simplify the reading and understanding of the RFC. + ### Miscellaneous endpoints #### get_status @@ -1031,18 +1055,23 @@ or a block that doesn't exist: Like the v1 API, the v2 API will use basic auth with the same secret. This token is usually in `grin/main/.api_secret`. -### Wallet support +## Wallet support The wallet currently uses the Grin Node API v1 so changes will be needed in the grin-wallet repository to make it compatible with the v2 Node API. -### Legacy support +## Legacy support The v1 API will remain active for a time the mode of operation for its REST API will be assumed to work as currently. This setup should allow existing wallets and apps to continue working as-is until a cutoff release for legacy mode is determined. -### API only +## API only Note that this RFC does not propose making user-facing changes to the existing CLI (invoked by `grin client`) to invoke these functions. It's expected that the existing cli functionality will be modified to invoke the new API functions. +## Node V1 Depreciation Timeline + +- v3.0.0 (January 2020): Node API v1 is marked as deprecated and will be remove in next major version. +- v4.0.0 (July 2020): Node API v1 is removed from the code. + # Drawbacks [drawbacks]: #drawbacks From 84acfc0f8bc7ccd7c2f849bfc00962b5d213c790 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 16 Oct 2019 11:07:53 -0400 Subject: [PATCH 20/61] Replace miscelaneous by Node --- text/0000-node-api-v2.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 5541ebf..5fcba54 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -15,7 +15,7 @@ - [Reference-level explanation](#reference-level-explanation) * [Current Endpoints with the v1 API](#current-endpoints-with-the-v1-api) * [Proposed Endpoints](#proposed-endpoints) - + [Miscellaneous endpoints](#miscellaneous-endpoints) + + [Node endpoints](#node-endpoints) - [get_status](#get_status) - [get_version](#get_version) + [Chain endpoints](#chain-endpoints) @@ -182,7 +182,7 @@ The new endpoint methods are the following: NB: The following v2 endpoints are classified by categories solely to simplify the reading and understanding of the RFC. -### Miscellaneous endpoints +### Node endpoints #### get_status From ec0f64ff639792b2a643a13c479bcb4bb70ddc36 Mon Sep 17 00:00:00 2001 From: antiochp <30642645+antiochp@users.noreply.github.com> Date: Tue, 29 Oct 2019 14:09:13 +0000 Subject: [PATCH 21/61] enable faster sync --- text/0000-enable-faster-sync.md | 81 +++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 text/0000-enable-faster-sync.md diff --git a/text/0000-enable-faster-sync.md b/text/0000-enable-faster-sync.md new file mode 100644 index 0000000..6b8433e --- /dev/null +++ b/text/0000-enable-faster-sync.md @@ -0,0 +1,81 @@ + +- Title: enable-faster-sync +- Authors: [Antioch Peverell](mailto:apeverell@protonmail.com), [John Tromp](mailto:john.tromp@gmail.com) +- Start date: Oct 25, 2019 +- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) +- Tracking issue: [Edit if merged with link to tracking github issue] + +--- + +# Summary +[summary]: #summary + +This RFC introduces changes that reduce the time it takes for a node joining the Grin network to fully sync. Data can be downloaded in batches and partially verified without the bottleneck of waiting to receive all data. These batches can be requested from multiple peers in parallel and verified as they are received. This improves the performance and the robustness of the "fast sync" process and allows for futher improvements to be explored in the future. A consensus change is required to facilitate this, with a change to what we commit to in each block header. We propose making this consensus change as part of the upcoming scheduled hardfork (HF2 scheduled for Jan 2020). + +# Motivation +[motivation]: #motivation + +The initial sync process involves downloading all kernels, all unspent outputs and their corresponding rangeproofs. The unspent outputs are maintained in a "Prunable Merkle Mountain Range" (PMMR). Briefly, this consists of unpruned output data, a set of intermediate hashes representing pruned subtrees and a "leafset" bitmap representing the position of unpruned, unspent outputs in the PMMR. + +The previous implementation packaged all of the above data into a single zip file, requested and received from a single peer. This was both a performance bottleneck and a single point of failure. To increase reliability and performance we split this data up into smaller chunks and request from multiple peers. + +It was previously only possible to validate the _full_ set of unspent outputs. We were unable to validate a partial set of outputs as we compared the total output sum to the total kernel sum plus the total kernel offset. By design, there is no partial kernel sum to compare against a partial output sum. + +This RFC describes how we split the output PMMR into subtrees, requesting these subtrees along with the corresponding bitmap fragment and the necessary Merkle proofs. Each output subtree can be verified with its corresponding bitmap fragment. By committing to the leafset bitmap in addition to the output PMMR we can verify the output subtrees. + +The final "sum to zero" validation remains unchanged but the partial verification of the individual output subtrees allows for more granular, and earlier verification of the data as it is received. + +# Community-level explanation +[community-level-explanation]: #community-level-explanation + +Validation of full transaction history is not necessary when joining the Grin network and access to full block history is not required. The initial sync process involves downloading all block headers, all kernels, all unspent outputs and their corresponding rangeproofs. Historical spent outputs are not necessary for validation. Nodes prune historical spent outputs, with a few days of recent history retained. Kernels are not pruned and must be retained indefinitely. + +The unspent outputs are maintained in a "Prunable Merkle Mountain Range" (PMMR). Briefly, this consists of unpruned output data, a set of intermediate hashes representing pruned subtrees and a "leafset" bitmap representing the position of unpruned, unspent outputs in the PMMR. Subtrees of the output PMMR are requested from peers and these subtrees are verified together with the corresponding fragment of the leafset bitmap. Merkle proofs are provided to prove inclusion of both the output subtree and bitmap fragments beneath the root committed to in the block header. Once all subtrees are received and verified, the unspent outputs are summed together and checked against the sum of all kernels plus the kernel offset from the block header. + +This allows us to download outputs from multiple peers, in parallel and verify them as they are received. +The node does not need to wait for the full output PMMR to be received before validation can begin. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +The leafset bitmap is committed to by splitting the bitmap into smaller fragments, adding these fragments to an MMR and committing to the root of the bitmap MMR. + +The leafset bitmap represents leaf positions, in insertion order, of the underlying output PMMR. The output PMMR is append-only and the order of these outputs does not change. The first (leftmost) bit in the bitmap maps directly to the first output inserted into the output PMMR. The next bit in the bitmap maps to the second output inserted into the output PMMR etc. The bitmap `11100010` represents outputs `[0, 1, 2, 6]` in the PMMR. The bitmap is then split into smaller fragments of 1024 bits, representing contiguous sequences of bits of the bitmap. The two fragments `0000...0001, 1000...0000` represent outputs `[1023, 1024]` in the output PMMR. Every fragment is exactly 1024 bits in length with the final fragment padded with `0` values to fill the full 1024 bits. + +The bitmap MMR itself is not append-only as fragments of the bitmap will be updated over time as outputs are spent. Recent outputs are more likely to be spent than old output, resulting in updates clustering around the most recent (rightmost) fragments in the MMR. This has the nice property of minimizing the amount of updating and rehashing necessary to keep the bitmap MMR updated over time as older fragments will change only rarely. + +Merkle proofs can be provided for both a subtree of the output PMMR and a fragment in the bitmap MMR. These are used during validation to locate the output subtree in the overall output PMMR and to verify the bitmap fragment corresponds to the same output positions. + +To commit to the root of both the output PMMR and the root of the bitmap MMR we hash the roots together +and commit to this single composite root `H(output_root|bitmap_root)` in the the block header. +This allows us to reuse the existing `output_root` field on the header to commit to both MMR roots. + + +# Drawbacks +[drawbacks]: #drawbacks + +These changes necessitate a change to the data committed to in the block header. This is a consensus breaking change and comes with all risks and complexity inherent to any consensus breaking change. As block headers are retained indefinitely the implementation will need to maintain compatibility for headers pre and post hardfork indefinitely. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +We believe we cannot make the desired changes to fast sync without committing to the leafset bitmap. +The previous approach of implicitly validating this as part of the overall "sum to zero" kernel validation does not allow for a batched approach. The previous approach was by design "all or nothing" with validation only possible once we have downloaded the full set of kernels and unspent outputs. + +The details are around precisely how to commit to this bitmap and there are various alternatives possible here. +The leafset bitmap is implemented as a "roaring bitmap" internally but we want to commit to a canonical representation of this data. The obvious choice here is the uncompressed bitmap data itself. We could simply hash the full bitmap but this is not efficient as we would need to rehash all the data for every block header. The other extreme would be to treat each individual bit in the bitmap as a leaf of a Merkle tree. This would also be inefficent as we would require a 32 byte hash for every individual bit. The proposed solution is to break the bitmap up into smaller chunks and add these chunks as leaves of an MMR. Recent outputs are more likely to be spent than older outputs and bitmap chunks to the right of the MMR +are more likely to change over time. These rightmost chunks have shorter paths to the associated peak in the MMR with correspondingly shorter Merkle proofs. The cost of updating more recent fragments is also minimized as the number of rehashing operations up to the peak is small. + +The proposal is to break the bitmap up into 1024 bit (128 byte) chunks and use these as leaves in the bitmap MMR. + +The other question is what data to add to the header itself. We propose combining the bitmap root and the output MMR root into a single `output_root` hash on the block header. This changes the semantics of the existing `output_root` field. This solutiuon has the advantage of maintaining the existing header serialization. No new header field would be necessary. +An alternative solution would be to keep the two MMR roots separate and to introduce the new bitmap root as a new header field. The would affect serialization at both the p2p layer and the local database and would add significant complexity to the implementation. + +The two roots are closely related and we propose committing to them together as a single combined hash. + +# References +[references]: #references + +* [Merkle Mountain Range (OpenTimestamps)](https://github.com/opentimestamps/opentimestamps-server/blob/532033b465c3b09c9db2a9064de03230c7e2e28e/doc/merkle-mountain-range.md) + +* [Prunable Merkle Mountain Ranges (Grin)](https://github.com/mimblewimble/grin/blob/67057ab36d606072c543b2741e33496c6affd6ab/doc/mmr.md) From 329ac9a01eec7e50f07f6cf41ffffa0ed8c21464 Mon Sep 17 00:00:00 2001 From: Yeastplume Date: Mon, 4 Nov 2019 12:31:03 +0000 Subject: [PATCH 22/61] add wallet update process rfc --- text/0000-wallet-update-process.md | 168 +++++++++++++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 text/0000-wallet-update-process.md diff --git a/text/0000-wallet-update-process.md b/text/0000-wallet-update-process.md new file mode 100644 index 0000000..b2e680e --- /dev/null +++ b/text/0000-wallet-update-process.md @@ -0,0 +1,168 @@ + +- Title: wallet-update-process +- Authors: [Michael Cordner](mailto:yeastplume@protonmail.com) +- Start date: Nov 4th, 2019 +- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) +- Tracking issue: [mimblewimble/grin-wallet#244](https://github.com/mimblewimble/grin-wallet/issues/244) + +--- + +# Summary +[summary]: #summary + +The changes outlined in the RFC are intended to make process of updating a wallet's state from the chain more consistent for developers and more transparent to the end user. The following changes are proposed: + +* Modify the wallet's update process to be more consistent and encapsulated. +* Run the current `check-repair` process as part of normal wallet update operations +* Provide the ability to run the wallet update process on a separate thread. +* Incorporate a TTL (Time-to-Live) field in the Slate +* Change the default output selection method to `smallest` + +# Motivation +[motivation]: #motivation + +The grin wallet currently updates the state of its outputs and transactions using a combination of UTXO updates, Kernel Lookups and UTXO set scanning. These 'primitives' on their own are generally enough to keep wallet states consistent, however the manner in which they are currently invoked is less than ideal and relies heavily on manually invoking the `check-repair` process and `cancel` command. + +To rectify this situation, this RFC outlines changes to the wallet update process that have the goal of ensuring wallets are always in a consistent state in a manner that is transparent to the user. It is hoped that upon adopting these changes, most wallets will automatically keep themselves in sync with the chain in all cases, and that users will not usually have to invoke the `check-repair` or `cancel` commands. + +# Community-level explanation +[community-level-explanation]: #community-level-explanation + +While these changes should be transparent to end users, wallet developers should note the changes to how the wallet updates itself via these processes. Special attention should be paid to the new update thread API functions, as well as the new TTL field and default selection strategy change. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Current Wallet Update Procedures + +The following sections outline the current processes used in the creation and update of outputs and related transactions within the wallet. + +### Transaction and Output Creation + +#### Coinbase Output + Transaction Creation + +Coinbase outputs and their related transactions are created via the wallet listener as part of the mining process. The currrent workflow is: + + * When mining, create a potential coinbase output for the target block. + * If the block is accepted and the output is detected in the UTXO set (via the [Update by Output process](#)), create a transaction log entry of type 'Confirmed Coinbase', with 'confirmed' set to true. + +#### Transacting (Payer -> Payee) Output + Transaction Creation + +Outputs and tranasctions are created during the transaction exchange process as follows. Note that the invoicing workflow is mostly identical with the roles of Payer and Payee reversed, so this workflow is not outlined separately. + + * The sender creates a new 'blank' transaction Slate, adding inputs and change outputs to the slate. + * The sender generates a random kernel offset. (The kernel offset is always generated by the slate creator) + * The sender sends the slate to the payee (via file or http). If sending synchronously, (e.g via http) the associated transaction is saved to the log after a response from the payee's listener. If sending asynchronously, (e.g. via file), the transaction is saved immediately. + * When saved, the associated transaction is set to type 'Sent Tx' with a status of 'unconfirmed', inputs are locked internally and a change output is added with status 'unconfirmed'. + * The payee receives the slate, creates (an) output(s) for the received amount with status unconfirmed, and immediately stores a transaction in their log of type 'Received Tx' with confirmed set to false. + * The payee calculates and saves the transaction kernel for later reference. + * The slate is returned to the payer for completion, who calculates and saves the transaction kernel. The transaction is sent to a node for validation. + +### Transaction and Output Statue Update Processes + +#### Update by Output Process + +The Update by Output Process has been the main method by which the wallet updates its internal state, deciding whether a transaction has been confirmed and when to remove or confirm individual outputs. This process works as follows: + +* For every unspent output in the wallet: +* If the output's status in the user's wallet is 'unconfirmed' and it is present in the UTXO set, change the output's status to 'confirmed', and update the associated transaction status to 'confirmed'. +* If the output's status in the user's wallet is 'confirmed' and it no longer appears in the UTXO set, set it's status to 'Spent'. Note these outputs will usually be locked so they cannot be selected for spending again. + +#### Kernel Lookup Process + +It is also possible to look up a transaction via the kernel that was stored when the transaction and outputs were created. This is necessary in cases such as where a participant in a transaction doesn't have any change outputs, which means the output update process won't detect when to mark a transaction confirmed. This process is: + +* Retrieve the transction kernel from the node using the kernel excess value calculated during the transaction creation process. +* If the kernel exists, update the status of the associated transaction to 'Confirmed' + +#### Check-Repair Process + +The Update by Output process on its own is not enough to ensure the contents of a wallet is correct. There are many easily-encountered situations in the course of a wallet's operation where this process is insufficient and can potentially lead to an wallet state that's inconsistent with the UTXO set, including but not limited to: + +* Manually cancelling transctions and unlocking outputs before they've had a chance to confirm on the chain. +* Running multiple wallets from the same seed +* Fork situations + +The check-repair process fixes most of these potential issues by scanning every single output in the UTXO set and testing for ownership by determining if its bullet proof can be decoded using the wallet's master key. It will then 'check and repair' all outputs and transactions in the wallet as follows: + +* If an output exists in the UTXO set and is not in the wallet, create an output for the wallet at the derivation path stored in the bullet proof, and create a new transaction crediting the amount +* If an output exists in the UTXO set and is marked as 'spent', mark it as unspent and cancel any associated transaction log entries. + +Additionally, the check-repair process can take a flag instructing it to delete unconfirmed outputs and reset all outstanding transctions. If this flag is set: + +* If a locked output exists in the UTXO set, unlock it and set any associated transactions to `cancelled` +* If an 'unconfirmed' output is not in the UTXO set, delete it and cancel any associated transction log entries. + +Note that the wallet `restore` process works very similarly to the check-repair process, however it must always operate on an empty wallet meaning it is limited to creating outputs and transactions in the wallet. + +## Current Overall Update Process + +All of the procedures outlined above can be considered set of 'primitives' available for the wallet to keep itself updated. The current update process is somewhat limited, and has the tendency to require a significant amount of manual updating using the `check-repair` command. The process today is as follows: + +* Invoke the 'Update by Output' process before retrieving any info relating to the wallet state (`txs`, `info` or `outputs` commands,) or creating a new transaction. This invocation is usually done internally and in a syncronous blocking manner by each command that requires the wallet state to be as up to date as possible. +* During the `txs` retrieval command, for each unconfirmed transaction with an `amount recieved` field set to 0, invoke the 'Kernel Lookup' process. +* If the user suspects the wallet state to be inconsistent with the node's UTXO state, the user can run the manual `check-repair` command. This will scan all UTXOs in the set from position 1 in the output PMMR. The user can optionally provide a flag to remove all unconfirmed transactions. +* If a transaction and associated outputs appear 'stuck' or 'locked' due to the other party not completing a transaction, the user must manually cancel the transactions and unlock the outputs for re-use. (or provide the `cancel-unconfirmed` flag to the check-repair process) +* The default selection strategy is to 'sweep' the wallet of outputs on each transaction creation, meaning that by default, no further transaction can usually be made while a transaction is outstanding. + +## Changes to Overall Update Process and Procedures + +While a combination of the 'primitives' listed above should be enough to keep a wallet's state consistent with a node's UTXO set, the current method of invoking the vital `check-repair` command is manual, and must be run on the entire UTXO each time. The overall update process will be modified to incorporate the current `check-repair` logic as part of normal operation. To ensure a wallet is only scanning the part of the UTXO set required, the wallet will store details on what parts of the UTXO set it has already scanned, and perform scans incrementally as part of normal update operation. + +### Rename `check-repair` to `scan` + +The naming of current `check-repair` command and functionality implies something has gone wrong, whereas it really should be considered a necessary part of normal wallet operation. It will therefore be renamed to a more friendly-sounding `scan` process. + +### New Overall Update Process + +The Overall Update process will proceed as follows: + +* Perform the 'Update by Output' process. +* Perform the 'Kernel Update' process for transactions where the incoming amount is 0. +* Query the wallet's internal data for the last block height scanned via the `scan` process. + * If the stored block's header hash doesn't match the version currently on the node's UTXO set, scan the range of the UTXO set corresponding to the last stored block - 100 blocks up until the current block. + * If the stored block's header matches the version on the chain, scan a range of the UTXO set corresponding to the stored block height to the current block height. +* Save the last block scanned by the `scan` process. + +### New / Recovered Wallets + +Newly created and recovered wallets will scan the UTXO set as follows: + +* New wallets generated with a new random seed will mark themselves as `new`, and set their `last scanned block` to the block height reported on first successful contact with a node. +* Wallets recovered with an existing mnemonic will set the `last scanned block` to `1`, triggering the `scan` process and thereby restoring any associated outputs. +* The `restore` command will be removed in favor of the `scan` process. + +### Manual Scans + +It will still be possible to manually scan the chain via the `scan` (`check`) command, however the command will be modified to allow the user to specify a range of block heights to scan. If this is not provided, the scan will occur from the current block - 1 week's worth of blocks. + +### Invocation of Update Process + +The update process is currently run inline in a synchronous blocking fashion as part of individual wallet commands. With the `scan` command becoming part of normal operations, it's expected that a particlar invocation the overall update process could potentially take a long time. This is acceptable in single-use modes such as the current single invocation model of `grin-wallet`, but will be far less usable in environments where the wallet and Owner API stay resident and the caller may not have any particular insight as to why a call invoking the update process might be taking a long time to complete. + +To address this, the wallet will provide a method of calling the Overall Update Process in its own thread that will attempt to mimimise its usage of the wallet lock. The Wallet's Owner API (V3 Only) will be extended with the following functions: + +* `start_update_thread(interval)` => Starts the update thread, calling the Overall Update Process at the frequency specified +* `stop_update_thread()` => Stops the update thread +* `get_update_status()` => Returns the current status of the update thread, along with as much information as possible about what the thread is doing and, in the case of the `scan` process, the percentage complete. + +Further, if the update thread is currently running, commands that currently call the Overall Update Process (such as `info`, `txs`) etc will not call the update process. This means that when invoked via the current command line, the behaviour should be unchanged. However, wallets running the Owner API can instead choose to run and monitor the background update thread at the desired frequency while keeping the user informed of any long-running update processes. + +### Transaction TTL + +The Update process outlined above does not cover the case where a slate is exchanged and outputs are locked, but one party doesn't complete the transaction for whatever reason. + +Currently, a situation such as this means that the outputs associated with a transaction sit locked indefinitely within a user's wallet, meaning that the only way to recover the outputs is with a manual `cancel` command (or a `check-repair` with the `delete-unconfirmed` flag set). Worse, the wallet's default selection strategy is currently set to `all` meaning an entire wallet's contents become locked on each transaction, meaning a single uncompleted transaction will render a user's wallet unusable. + +To rectify these problems, the following changes will be made: + +* The addition of a Time-To-Live (`TTL`) field to the Slate, which will be defined as the last block height at which a wallet should attempt to complete a transaction. This field should be respected at the wallet-exchange level, but is not currently commit to at a consensus level. If a wallet detects a particlar transaction's TTL has expired, it will automatically cancel the transaction and unlock associated outputs. Note this doesn't prevent the other party sending the transaction to the node after this happens, but there is no guarantee the outputs will still be available. However, if this does happen the `scan` command will correct the wallet state. +* Change the default output selection method to `smallest`, to prevent all wallet amounts being locked on every transaction. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- Do we limit the Kernel Lookup part of the process to just cases where there is no change output. Is there any benefit to calling it for all outstanding transactions on each lookup (or any particular downside to doing so?) +- Can we add the TTL field at consensus level? Making it consensus means much less scope for inconsistency between wallets overall, but how would it be commit to? (Could it be inclded in the kernel excess somehow, for example?) +- What is the effect on UTXO set size of changing the selection method to `smallest`? +- Confirm sensible defaults for how far to scan back on a `scan` \ No newline at end of file From f4fde30c5b0412dae0b991961f6bc887d4e735a5 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Tue, 5 Nov 2019 09:55:13 -0500 Subject: [PATCH 23/61] Create 0000-payment-proofs.md --- text/0000-payment-proofs.md | 64 +++++++++++++++++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 text/0000-payment-proofs.md diff --git a/text/0000-payment-proofs.md b/text/0000-payment-proofs.md new file mode 100644 index 0000000..99ea8b7 --- /dev/null +++ b/text/0000-payment-proofs.md @@ -0,0 +1,64 @@ +- Title: payment-proofs +- Authors: [David Burkett](mailto:davidburkett38@gmail.com) + +--- + +# Summary +[summary]: #summary + +Support generating and validating payment proofs for sender-initiated (ie. non-invoice) transactions. + +# Motivation +[motivation]: #motivation + +Bitcoin and other cryptocurrencies with transparent protocol-level addressing and immutable, unpruneable blockchains can prove sender, receiver, and amounts of payments simply by pointing to the transaction in the blockchain. +Grin's privacy and scalability means users no longer have this ability. This prevents some merchants from accepting Grin due to the high possibility of payment disputes that are unresolvable in the same way they are for transparent coins. + +This RFC proposes a change to the transaction building process where payers can require payees to create a "proof" they've received a payment before the payer finalizes and broadcasts the transaction. + +# Community-level explanation +[community-level-explanation]: #community-level-explanation + +From an end-user perspective, payers can require payees to prove receipt of funds as part of the transacting process. +Payers can then use these "proofs", along with information from the blockchain, to resolve payment disputes and prove funds were sent to the correct payee, and the transaction was confirmed on the blockchain. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Common workflow + +## Slate changes + +## Generating proofs + +## Verifying proofs + +## Wallet actions +### init-send + +### receive + +### finalize + +# Drawbacks +[drawbacks]: #drawbacks + +* Drawback 1 +* Drawback 2 + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +* Can this be adapted to work for invoices? +* Question 2 + +# Future possibilities +[future-possibilities]: #future-possibilities + +None + +# References +[references]: #references + +None + From e170da0bce97c2843b3e233cd85e65ca6caed7b6 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Tue, 12 Nov 2019 05:37:35 -0500 Subject: [PATCH 24/61] Update 0000-payment-proofs.md --- text/0000-payment-proofs.md | 52 ++++++++++++++++++++++++++++++++----- 1 file changed, 45 insertions(+), 7 deletions(-) diff --git a/text/0000-payment-proofs.md b/text/0000-payment-proofs.md index 99ea8b7..fb25150 100644 --- a/text/0000-payment-proofs.md +++ b/text/0000-payment-proofs.md @@ -25,32 +25,69 @@ Payers can then use these "proofs", along with information from the blockchain, # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -## Common workflow - ## Slate changes +A new, optional, structure (`payment_info`) will be added to transaction slates, along with a version increase. The structure will contain: + +* `message` - Optional message to include in the payment proof. Limited to 100(?) characters. +* `sender_address` - Optional ed25519 public key generated by the sender. +* `receiver_address` - An ed25519 public key for the receiver, typically the public key of the user's v3 onion address. +* `receiver_signature` - A signature of the sender_address, received amount, and kernel commitment that validates against the `receiver_address`. + ## Generating proofs -## Verifying proofs +Receipt confirmations (`receiver_signature`s) will be generated by the payee by providing an ED25519 signature of: `(amount || kernel_commitment || sender_address || message)`, using the private key of the `receiver_address`. + +If a `sender_address` was provided, then `sender_signature` can be generated for `(amount || kernel_commitment || sender_address || message)` using the private key of the `sender_address`. + +Sender will then create and store the following info, which can be considered the complete `payment_proof`: + +* `receiver_address` +* `receiver_signature` +* `amount` +* `kernel_commitment` +* `sender_address` (OPTIONAL) +* `sender_signature` (OPTIONAL) +* `message` (OPTIONAL) + +## Verifying Proofs + +This `payment_proof` can be provided by the sender at any time to convince a payee that a payment was made to them. The proof can be verified as follows: + +1. Ensure the `kernel_commitment` is confirmed on-chain. +2. Verify that the `receiver_address` belongs to the payee. +3. Verify that the `receiver_signature` is valid. +4. If a `sender_address` was included, verify that a valid `sender_signature` was provided. ## Wallet actions ### init-send +As part of the first step of the tx-building process, the sender/payer can optionally choose to generate the `sender_address` using their keychain. + +The `receiver_address`, optional keychain path of the `sender_address`, and the optional `message` must be stored locally, along with the `slate_id`. + +The `sender_address`, `receiver_address`, and `messsage` will then be added to the `payment_proof` structure of the slate. + ### receive +If the `payment_proof` structure exists on the slate, it is mandatory that the `receiver_signature` is generated and added to the slate as part of the receive tx-building step. + ### finalize +Using the `slate_id`, the sender can retrieve the original `sender_address`, `receiver_address`, and `message` that were included in the slate, and verify that those fields remain unchanged. The sender must then validate the `receiver_signature`. If any of the original `payment_proof` slate fields were modified, or if the `receiver_signature` is invalid, the transaction must be rejected by the sender. + +Once the `payment_info` details have been validated, the sender can generate and store the `payment_proof` (See `Generating Proofs` above), and then finalize the transaction as normal. + # Drawbacks [drawbacks]: #drawbacks -* Drawback 1 -* Drawback 2 +None # Unresolved questions [unresolved-questions]: #unresolved-questions +* Should the `sender_address` and `sender_signature` be optional, required, or removed entirely? * Can this be adapted to work for invoices? -* Question 2 # Future possibilities [future-possibilities]: #future-possibilities @@ -60,5 +97,6 @@ None # References [references]: #references -None +* Tx slate structure: https://github.com/mimblewimble/grin-wallet/blob/master/libwallet/src/slate.rs +* Beam's approach: https://github.com/BeamMW/beam/blob/c9beb0eae55fa6b7fb3084ebe9b5db2850cf83b9/wallet/wallet_db.cpp#L3231-L3236 From b98b647f99bcb575569d5899a1f225ec137551d0 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Tue, 12 Nov 2019 05:40:58 -0500 Subject: [PATCH 25/61] Update 0000-payment-proofs.md --- text/0000-payment-proofs.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/text/0000-payment-proofs.md b/text/0000-payment-proofs.md index fb25150..a128a01 100644 --- a/text/0000-payment-proofs.md +++ b/text/0000-payment-proofs.md @@ -27,7 +27,7 @@ Payers can then use these "proofs", along with information from the blockchain, ## Slate changes -A new, optional, structure (`payment_info`) will be added to transaction slates, along with a version increase. The structure will contain: +A new (optional) structure (`payment_info`) will be added to transaction slates, along with a version increase. The `payment_info` structure will contain: * `message` - Optional message to include in the payment proof. Limited to 100(?) characters. * `sender_address` - Optional ed25519 public key generated by the sender. @@ -98,5 +98,4 @@ None [references]: #references * Tx slate structure: https://github.com/mimblewimble/grin-wallet/blob/master/libwallet/src/slate.rs -* Beam's approach: https://github.com/BeamMW/beam/blob/c9beb0eae55fa6b7fb3084ebe9b5db2850cf83b9/wallet/wallet_db.cpp#L3231-L3236 - +* Beam's payment proof model: https://github.com/BeamMW/beam/blob/c9beb0eae55fa6b7fb3084ebe9b5db2850cf83b9/wallet/wallet_db.cpp#L3231-L3236 From 8c848e95d80f84c9e33dadb3a597bd30e3d80f14 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Wed, 13 Nov 2019 18:18:30 -0500 Subject: [PATCH 26/61] Add get_pmmr_indices method --- text/0000-node-api-v2.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 5fcba54..4a32db6 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -585,9 +585,33 @@ UTXO traversal. Retrieves last utxos since a start index until a max. Last boole } } } +``` + +#### get_pmmr_indices + +Retrieves the PMMR indices based on the provided block height(s). +```JSON +{ + "jsonrpc": "2.0", + "method": "get_pmmr_indices", + "params": [1, 200], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": { + "highest_index": 398, + "last_retrieved_index": 2, + "outputs": [] + } + } +} ``` + #### validate_chain Trigger a validation of the chain state. From 0c55eaf38988ef1e469a3806d9c7eccf478f3c4e Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 14 Nov 2019 12:17:41 -0500 Subject: [PATCH 27/61] lehnberg comments --- text/0000-node-api-v2.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 4a32db6..b4696f9 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -44,7 +44,7 @@ * [Wallet support](#wallet-support) * [Legacy support](#legacy-support) * [API only](#api-only) - * [Node V1 Depreciation Timeline](#node-v1-depreciation-timeline) + * [Node V1 Deprecation Timeline](#node-v1-deprecation-timeline) - [Drawbacks](#drawbacks) - [Prior art](#prior-art) - [Future possibilities](#future-possibilities) @@ -60,13 +60,13 @@ Create a v2 JSON-RPC API for the Node API. [motivation]: #motivation -The current Node API (referred here as v1 API) is a REST API. This API while simple is: +The previous Node API (referred to here as v1) was a REST API. This API while simple had a few drawbacks: -- Documented manually (current documentation is available [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md). +- Manually documented (current documentation is available [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md). - Contains call with heterogenous args such as `?byid=xxx` and `commitment/xxx` which can be confusing and lack some uniformity. - Uses REST which is bound to HTTP while v2 wallet API uses JSON-RPC. -The goal of this RFC is to provide a new API with: +This RFC provides a new v2 API with: - Cleaner methods and errors. - Generated documentation directly on docs.rs. - Automatic testing with doc tests. @@ -82,9 +82,9 @@ This new API will be particularly useful for wallet developer and should ultimat [reference-level-explanation]: #reference-level-explanation -## Current Endpoints with the v1 API +## V1 Endpoints -While the endpoints are documented in details [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md), here is an overview of the current REST API Endpoints of the v1 API. +While the endpoints are documented in details [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md), here is an overview of the REST API Endpoints of the v1 API. ```JSON [ @@ -122,9 +122,9 @@ These endpoints can be grouped into 5 categories: - `pool` endpoints - `txhashset` endpoints -## Proposed Endpoints +## V2 Endpoints -This RFC proposes to keep the logic of the following endpoints and refactor them: +The following endpoints are kept and refactored: ```JSON [ @@ -1077,11 +1077,11 @@ or a block that doesn't exist: ## Authentication -Like the v1 API, the v2 API will use basic auth with the same secret. This token is usually in `grin/main/.api_secret`. +Like the v1 API, the v2 API uses basic auth with the same secret. This token is usually in `grin/main/.api_secret`. ## Wallet support -The wallet currently uses the Grin Node API v1 so changes will be needed in the grin-wallet repository to make it compatible with the v2 Node API. +For full compatibility, grin-wallet will migrate to use the v2 Node API. ## Legacy support @@ -1089,9 +1089,9 @@ The v1 API will remain active for a time the mode of operation for its REST API ## API only -Note that this RFC does not propose making user-facing changes to the existing CLI (invoked by `grin client`) to invoke these functions. It's expected that the existing cli functionality will be modified to invoke the new API functions. +Note that this RFC doe not make user-facing changes to the existing CLI (invoked by `grin client`) to invoke these functions. It's expected that the existing cli functionality will be modified to invoke the new API functions. -## Node V1 Depreciation Timeline +## Node V1 Deprecation Timeline - v3.0.0 (January 2020): Node API v1 is marked as deprecated and will be remove in next major version. - v4.0.0 (July 2020): Node API v1 is removed from the code. @@ -1100,7 +1100,7 @@ Note that this RFC does not propose making user-facing changes to the existing C [drawbacks]: #drawbacks -The implementation of this RFC will temporarily introduce some additional code complexity as v1 and v2 node API will need to coexist until a cutoff release is determined. +This RFC temporarily introduces some additional code complexity as v1 and v2 node API will need to coexist for the duration of the deprecation period of v1. # Prior art @@ -1117,7 +1117,7 @@ This kind of JSON-RPC API is widely used for cryptocurrencies. For instance: [future-possibilities]: #future-possibilities -This new API will simplify the deployment of new methods and will drastically simplify the work of developers by providing a clear documentation directly on docs.rs. +This API simplifies the deployment of new methods and drastically simplifies the work of developers by providing a clear documentation directly on docs.rs. # References From 5c36ecace6fbbed7c838b68c98c2ee6d8d4cff9f Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 14 Nov 2019 12:18:45 -0500 Subject: [PATCH 28/61] Add missing link in TOC --- text/0000-node-api-v2.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index b4696f9..97f51dc 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -25,6 +25,7 @@ - [get_kernel](#get_kernel) - [get_outputs](#get_outputs) - [get_unspent_outputs](#get_unspent_outputs) + - [get_pmmr_indices](#get_pmmr_indices) - [validate_chain](#validate_chain) - [compact_chain](#compact_chain) + [Peer endpoints](#peer-endpoints) From 5a59f6c8817a30b166c6553e45de9554a3dc2198 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Thu, 14 Nov 2019 14:38:01 -0500 Subject: [PATCH 29/61] Addressing comments --- text/0000-payment-proofs.md | 44 +++++++++++++++++++++++-------------- 1 file changed, 28 insertions(+), 16 deletions(-) diff --git a/text/0000-payment-proofs.md b/text/0000-payment-proofs.md index a128a01..e66fc5e 100644 --- a/text/0000-payment-proofs.md +++ b/text/0000-payment-proofs.md @@ -1,26 +1,28 @@ - Title: payment-proofs - Authors: [David Burkett](mailto:davidburkett38@gmail.com) - +- Start date: Nov 5, 2019 +- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) +- Tracking issue: [Edit if merged with link to tracking github issue] --- # Summary [summary]: #summary -Support generating and validating payment proofs for sender-initiated (ie. non-invoice) transactions. +Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. # Motivation [motivation]: #motivation -Bitcoin and other cryptocurrencies with transparent protocol-level addressing and immutable, unpruneable blockchains can prove sender, receiver, and amounts of payments simply by pointing to the transaction in the blockchain. +Bitcoin and other cryptocurrencies with transparent protocol-level addressing and immutable, unprunable blockchains can prove sender, receiver, and amounts of payments simply by pointing to the transaction in the blockchain. Grin's privacy and scalability means users no longer have this ability. This prevents some merchants from accepting Grin due to the high possibility of payment disputes that are unresolvable in the same way they are for transparent coins. -This RFC proposes a change to the transaction building process where payers can require payees to create a "proof" they've received a payment before the payer finalizes and broadcasts the transaction. +This RFC changes the transaction building process where payers can require payees to create a "proof" they've received a payment before the payer finalizes and broadcasts the transaction. # Community-level explanation [community-level-explanation]: #community-level-explanation From an end-user perspective, payers can require payees to prove receipt of funds as part of the transacting process. -Payers can then use these "proofs", along with information from the blockchain, to resolve payment disputes and prove funds were sent to the correct payee, and the transaction was confirmed on the blockchain. +Payers can then use these "proofs" to resolve payment disputes and prove they sent funds to the correct payee. # Reference-level explanation [reference-level-explanation]: #reference-level-explanation @@ -30,15 +32,15 @@ Payers can then use these "proofs", along with information from the blockchain, A new (optional) structure (`payment_info`) will be added to transaction slates, along with a version increase. The `payment_info` structure will contain: * `message` - Optional message to include in the payment proof. Limited to 100(?) characters. -* `sender_address` - Optional ed25519 public key generated by the sender. +* `sender_address` - An ed25519 public key generated by the sender. * `receiver_address` - An ed25519 public key for the receiver, typically the public key of the user's v3 onion address. * `receiver_signature` - A signature of the sender_address, received amount, and kernel commitment that validates against the `receiver_address`. ## Generating proofs -Receipt confirmations (`receiver_signature`s) will be generated by the payee by providing an ED25519 signature of: `(amount || kernel_commitment || sender_address || message)`, using the private key of the `receiver_address`. +Receipt confirmations (`receiver_signature`) will be generated by the payee by providing an ED25519 signature of: `(amount || kernel_commitment || sender_address || message)`, using the private key of the `receiver_address`. -If a `sender_address` was provided, then `sender_signature` can be generated for `(amount || kernel_commitment || sender_address || message)` using the private key of the `sender_address`. +The `sender_signature` can be generated for `(amount || kernel_commitment || sender_address || message)` using the private key of the `sender_address`. Sender will then create and store the following info, which can be considered the complete `payment_proof`: @@ -46,8 +48,8 @@ Sender will then create and store the following info, which can be considered th * `receiver_signature` * `amount` * `kernel_commitment` -* `sender_address` (OPTIONAL) -* `sender_signature` (OPTIONAL) +* `sender_address` +* `sender_signature` * `message` (OPTIONAL) ## Verifying Proofs @@ -57,14 +59,14 @@ This `payment_proof` can be provided by the sender at any time to convince a pay 1. Ensure the `kernel_commitment` is confirmed on-chain. 2. Verify that the `receiver_address` belongs to the payee. 3. Verify that the `receiver_signature` is valid. -4. If a `sender_address` was included, verify that a valid `sender_signature` was provided. +4. Verify that the `sender_signature` is valid. ## Wallet actions ### init-send -As part of the first step of the tx-building process, the sender/payer can optionally choose to generate the `sender_address` using their keychain. +As part of the first step of the tx-building process, the sender/payer generates the `sender_address` using their keychain. -The `receiver_address`, optional keychain path of the `sender_address`, and the optional `message` must be stored locally, along with the `slate_id`. +The `receiver_address`, keychain path of the `sender_address`, and the optional `message` must be stored locally, along with the `slate_id`. The `sender_address`, `receiver_address`, and `messsage` will then be added to the `payment_proof` structure of the slate. @@ -81,18 +83,28 @@ Once the `payment_info` details have been validated, the sender can generate and # Drawbacks [drawbacks]: #drawbacks -None +* Increases the size of tx slates. +* Possibility of privacy leakage through address reuse. + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +* This design works well with TOR tx building, yet is generic enough to work with all known transacting mechanisms. + +# Prior art +[prior-art]: #prior-art + +* Wallet713 implements payment proofs for grinbox transactions, which our design adapts and builds on to work more seemlessly with onion addresses and with transaction building methods that don't inherently rely on addresses. # Unresolved questions [unresolved-questions]: #unresolved-questions -* Should the `sender_address` and `sender_signature` be optional, required, or removed entirely? * Can this be adapted to work for invoices? # Future possibilities [future-possibilities]: #future-possibilities -None +* Payment proofs could potentially be added to invoice payments in the future, but at the cost of an additional round of communication. # References [references]: #references From 715cfce662d24a1090ee2949574058d12f2982ef Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Fri, 15 Nov 2019 10:09:41 -0500 Subject: [PATCH 30/61] Add clear mention of the endpoint --- text/0000-node-api-v2.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index 97f51dc..fcedff1 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -183,6 +183,8 @@ The new endpoint methods are the following: NB: The following v2 endpoints are classified by categories solely to simplify the reading and understanding of the RFC. +When running `grin` with defaults, the V2 api is available at `localhost:3413/v2`. + ### Node endpoints #### get_status From 22d3835de1ec816de765c29ea4c3b5559d86a1b6 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Wed, 20 Nov 2019 06:34:11 -0500 Subject: [PATCH 31/61] Removing message field --- text/0000-payment-proofs.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/text/0000-payment-proofs.md b/text/0000-payment-proofs.md index e66fc5e..44b270a 100644 --- a/text/0000-payment-proofs.md +++ b/text/0000-payment-proofs.md @@ -31,16 +31,15 @@ Payers can then use these "proofs" to resolve payment disputes and prove they se A new (optional) structure (`payment_info`) will be added to transaction slates, along with a version increase. The `payment_info` structure will contain: -* `message` - Optional message to include in the payment proof. Limited to 100(?) characters. * `sender_address` - An ed25519 public key generated by the sender. * `receiver_address` - An ed25519 public key for the receiver, typically the public key of the user's v3 onion address. * `receiver_signature` - A signature of the sender_address, received amount, and kernel commitment that validates against the `receiver_address`. ## Generating proofs -Receipt confirmations (`receiver_signature`) will be generated by the payee by providing an ED25519 signature of: `(amount || kernel_commitment || sender_address || message)`, using the private key of the `receiver_address`. +Receipt confirmations (`receiver_signature`) will be generated by the payee by providing an ED25519 signature of: `(amount || kernel_commitment || sender_address)`, using the private key of the `receiver_address`. -The `sender_signature` can be generated for `(amount || kernel_commitment || sender_address || message)` using the private key of the `sender_address`. +The `sender_signature` can be generated for `(amount || kernel_commitment || sender_address)` using the private key of the `sender_address`. Sender will then create and store the following info, which can be considered the complete `payment_proof`: @@ -50,7 +49,6 @@ Sender will then create and store the following info, which can be considered th * `kernel_commitment` * `sender_address` * `sender_signature` -* `message` (OPTIONAL) ## Verifying Proofs @@ -66,9 +64,9 @@ This `payment_proof` can be provided by the sender at any time to convince a pay As part of the first step of the tx-building process, the sender/payer generates the `sender_address` using their keychain. -The `receiver_address`, keychain path of the `sender_address`, and the optional `message` must be stored locally, along with the `slate_id`. +The `receiver_address` and keychain path of the `sender_address` must be stored locally, along with the `slate_id`. -The `sender_address`, `receiver_address`, and `messsage` will then be added to the `payment_proof` structure of the slate. +The `sender_address` and `receiver_address` will then be added to the `payment_proof` structure of the slate. ### receive @@ -76,7 +74,7 @@ If the `payment_proof` structure exists on the slate, it is mandatory that the ` ### finalize -Using the `slate_id`, the sender can retrieve the original `sender_address`, `receiver_address`, and `message` that were included in the slate, and verify that those fields remain unchanged. The sender must then validate the `receiver_signature`. If any of the original `payment_proof` slate fields were modified, or if the `receiver_signature` is invalid, the transaction must be rejected by the sender. +Using the `slate_id`, the sender can retrieve the original `sender_address` and `receiver_address` that were included in the slate, and verify that those fields remain unchanged. The sender must then validate the `receiver_signature`. If any of the original `payment_proof` slate fields were modified, or if the `receiver_signature` is invalid, the transaction must be rejected by the sender. Once the `payment_info` details have been validated, the sender can generate and store the `payment_proof` (See `Generating Proofs` above), and then finalize the transaction as normal. From c725f12beb3b26557ce5862cacad08a1e1185a87 Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 28 Nov 2019 17:41:17 +0100 Subject: [PATCH 32/61] Add Owner and Foreign API --- text/0000-node-api-v2.md | 503 ++++++++++++++++++++------------------- 1 file changed, 253 insertions(+), 250 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index fcedff1..a031013 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -15,10 +15,16 @@ - [Reference-level explanation](#reference-level-explanation) * [Current Endpoints with the v1 API](#current-endpoints-with-the-v1-api) * [Proposed Endpoints](#proposed-endpoints) - + [Node endpoints](#node-endpoints) + + [Owner API Endpoints](#owner-api-endpoints) - [get_status](#get_status) + - [validate_chain](#validate_chain) + - [compact_chain](#compact_chain) + - [get_peers](#get_peers) + - [get_connected_peers](#get_connected_peers) + - [ban_peer](#ban_peer) + - [unban_peer](#unban_peer) + + [Foreign API Endpoints](#foreign-api-endpoints) - [get_version](#get_version) - + [Chain endpoints](#chain-endpoints) - [get_header](#get_header) - [get_block](#get_block) - [get_tip](#get_tip) @@ -26,14 +32,6 @@ - [get_outputs](#get_outputs) - [get_unspent_outputs](#get_unspent_outputs) - [get_pmmr_indices](#get_pmmr_indices) - - [validate_chain](#validate_chain) - - [compact_chain](#compact_chain) - + [Peer endpoints](#peer-endpoints) - - [get_peers](#get_peers) - - [get_connected_peers](#get_connected_peers) - - [ban_peer](#ban_peer) - - [unban_peer](#unban_peer) - + [Pool endpoints](#pool-endpoints) - [get_pool_size](#get_pool_size) - [get_stempool_size](#get_stempool_size) - [get_unconfirmed_transactions](#get_unconfirmed_transactions) @@ -66,9 +64,11 @@ The previous Node API (referred to here as v1) was a REST API. This API while si - Manually documented (current documentation is available [here](https://github.com/mimblewimble/grin/blob/master/doc/api/node_api.md). - Contains call with heterogenous args such as `?byid=xxx` and `commitment/xxx` which can be confusing and lack some uniformity. - Uses REST which is bound to HTTP while v2 wallet API uses JSON-RPC. +- No difference between node management and simple information endpoints (i.e. exposing the node on the internet would allow anyone to query sensitive endpoints) This RFC provides a new v2 API with: - Cleaner methods and errors. +- Owner and Foreign API. - Generated documentation directly on docs.rs. - Automatic testing with doc tests. - Stronger basis for future improvements. @@ -160,9 +160,18 @@ The logic of the following endpoints will NOT be implemented as they are purely "get txhashset/merkleproof?n=1", ``` -The new endpoint methods are the following: +The new Owner API endpoint methods are the following: - [get_status](#get_status) +- [validate_chain](#validate_chain) +- [compact_chain](#compact_chain) +- [get_peers](#get_peers) +- [get_connected_peers](#get_connected_peers) +- [ban_peer](#ban_peer) +- [unban_peer](#unban_peer) + +The new Foreign API endpoint methods are the following: + - [get_version](#get_version) - [get_header](#get_header) - [get_block](#get_block) @@ -170,22 +179,16 @@ The new endpoint methods are the following: - [get_kernel](#get_kernel) - [get_outputs](#get_outputs) - [get_unspent_outputs](#get_unspent_outputs) -- [validate_chain](#validate_chain) -- [compact_chain](#compact_chain) -- [get_peers](#get_peers) -- [get_connected_peers](#get_connected_peers) -- [ban_peer](#ban_peer) -- [unban_peer](#unban_peer) - [get_pool_size](#get_pool_size) - [get_stempool_size](#get_stempool_size) - [get_unconfirmed_transactions](#get_unconfirmed_transactions) - [push_transaction](#push_transaction) -NB: The following v2 endpoints are classified by categories solely to simplify the reading and understanding of the RFC. - -When running `grin` with defaults, the V2 api is available at `localhost:3413/v2`. +When running `grin` with defaults, the V2 apis are available at +- `localhost:3413/v2/owner` for the owner API. +- `localhost:3413/v2/foreign` for the foreign API. -### Node endpoints +### Owner API Endpoints #### get_status @@ -222,6 +225,232 @@ Returns various information about the node, the network and the current sync sta } ``` +#### validate_chain + +Trigger a validation of the chain state. + +```JSON +{ + "jsonrpc": "2.0", + "method": "validate_chain", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` + +#### compact_chain + +Trigger a compaction of the chain state to regain storage space. + +```JSON +{ + "jsonrpc": "2.0", + "method": "compact_chain", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` + + +### Peer endpoints + +#### get_peers + +Retrieves information about peers. If `null` is provided, `get_peers` will list all stored peers. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_peers", + "params": ["70.50.33.130:3414"], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": [ + { + "addr": "70.50.33.130:3414", + "ban_reason": "None", + "capabilities": { + "bits": 15 + }, + "flags": "Defunct", + "last_banned": 0, + "last_connected": 1570129317, + "user_agent": "MW/Grin 2.0.0" + } + ] + } +} +``` + +#### get_connected_peers + +Retrieves a list of all connected peers. + +```JSON +{ + "jsonrpc": "2.0", + "method": "get_connected_peers", + "params": [], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": [ + { + "addr": "35.176.195.242:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "47.97.198.21:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "148.251.16.13:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "68.195.18.155:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "52.53.221.15:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 0, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "109.74.202.16:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "121.43.183.180:3414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + }, + { + "addr": "35.157.247.209:23414", + "capabilities": { + "bits": 15 + }, + "direction": "Outbound", + "height": 374510, + "total_difficulty": 1133954621205750, + "user_agent": "MW/Grin 2.0.0", + "version": 1 + } + ] + } +} +``` + +#### ban_peer + +Bans a specific peer. + +```JSON +{ + "jsonrpc": "2.0", + "method": "ban_peer", + "params": ["70.50.33.130:3414"], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` + +#### unban_peer + +Unbans a specific peer. + +```JSON +{ + "jsonrpc": "2.0", + "method": "unban_peer", + "params": ["70.50.33.130:3414"], + "id": 1 +} +{ + "id": 1, + "jsonrpc": "2.0", + "result": { + "Ok": null + } +} +``` + +### Foreign API Endpoints + #### get_version Returns the node version and block header version (used by grin-wallet). @@ -245,8 +474,6 @@ Returns the node version and block header version (used by grin-wallet). } ``` -### Chain endpoints - #### get_header Gets block header given either a height, a hash or an unspent output commitment. @@ -614,232 +841,6 @@ Retrieves the PMMR indices based on the provided block height(s). } ``` - -#### validate_chain - -Trigger a validation of the chain state. - -```JSON -{ - "jsonrpc": "2.0", - "method": "validate_chain", - "params": [], - "id": 1 -} -{ - "id": 1, - "jsonrpc": "2.0", - "result": { - "Ok": null - } -} -``` - -#### compact_chain - -Trigger a compaction of the chain state to regain storage space. - -```JSON -{ - "jsonrpc": "2.0", - "method": "compact_chain", - "params": [], - "id": 1 -} -{ - "id": 1, - "jsonrpc": "2.0", - "result": { - "Ok": null - } -} -``` - -### Peer endpoints - -#### get_peers - -Retrieves information about peers. If `null` is provided, `get_peers` will list all stored peers. - -```JSON -{ - "jsonrpc": "2.0", - "method": "get_peers", - "params": ["70.50.33.130:3414"], - "id": 1 -} -{ - "id": 1, - "jsonrpc": "2.0", - "result": { - "Ok": [ - { - "addr": "70.50.33.130:3414", - "ban_reason": "None", - "capabilities": { - "bits": 15 - }, - "flags": "Defunct", - "last_banned": 0, - "last_connected": 1570129317, - "user_agent": "MW/Grin 2.0.0" - } - ] - } -} -``` - -#### get_connected_peers - -Retrieves a list of all connected peers. - -```JSON -{ - "jsonrpc": "2.0", - "method": "get_connected_peers", - "params": [], - "id": 1 -} -{ - "id": 1, - "jsonrpc": "2.0", - "result": { - "Ok": [ - { - "addr": "35.176.195.242:3414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 374510, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - }, - { - "addr": "47.97.198.21:3414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 374510, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - }, - { - "addr": "148.251.16.13:3414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 374510, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - }, - { - "addr": "68.195.18.155:3414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 374510, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - }, - { - "addr": "52.53.221.15:3414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 0, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - }, - { - "addr": "109.74.202.16:3414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 374510, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - }, - { - "addr": "121.43.183.180:3414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 374510, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - }, - { - "addr": "35.157.247.209:23414", - "capabilities": { - "bits": 15 - }, - "direction": "Outbound", - "height": 374510, - "total_difficulty": 1133954621205750, - "user_agent": "MW/Grin 2.0.0", - "version": 1 - } - ] - } -} -``` - -#### ban_peer - -Bans a specific peer. - -```JSON -{ - "jsonrpc": "2.0", - "method": "ban_peer", - "params": ["70.50.33.130:3414"], - "id": 1 -} -{ - "id": 1, - "jsonrpc": "2.0", - "result": { - "Ok": null - } -} -``` - -#### unban_peer - -Unbans a specific peer. - -```JSON -{ - "jsonrpc": "2.0", - "method": "unban_peer", - "params": ["70.50.33.130:3414"], - "id": 1 -} -{ - "id": 1, - "jsonrpc": "2.0", - "result": { - "Ok": null - } -} -``` - -### Pool endpoints - #### get_pool_size Returns the number of transactions in the transaction pool. @@ -1080,7 +1081,9 @@ or a block that doesn't exist: ## Authentication -Like the v1 API, the v2 API uses basic auth with the same secret. This token is usually in `grin/main/.api_secret`. +Like the v1 API, the v2 API uses basic auth. However, the foreign and owner API do not share the same secret. +- The Owner API use the same token as the V1 Rest API, usually defined in `grin/main/.api_secret`. +- The Foreign API use its own token, usually defined in `grin/main/.foreign_api_secret`. ## Wallet support From 0224bb6d7a9fe94070b47c3757039ef543b9816f Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 28 Nov 2019 17:42:13 +0100 Subject: [PATCH 33/61] Fix index --- text/0000-node-api-v2.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index a031013..b7344f3 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -265,9 +265,6 @@ Trigger a compaction of the chain state to regain storage space. } ``` - -### Peer endpoints - #### get_peers Retrieves information about peers. If `null` is provided, `get_peers` will list all stored peers. From 3ea1ae9844b694aa9b8b9e890642b2d25323b265 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Fri, 29 Nov 2019 02:32:44 -0500 Subject: [PATCH 34/61] Addressing PR comments. --- text/0000-online-transacting-via-tor.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/text/0000-online-transacting-via-tor.md b/text/0000-online-transacting-via-tor.md index 4ac2d09..6fce219 100644 --- a/text/0000-online-transacting-via-tor.md +++ b/text/0000-online-transacting-via-tor.md @@ -6,7 +6,7 @@ Start date: September 1, 2019
# Summary Describes a standardized addressing and communication protocol for building Grin transactions.

Wallets will attempt to connect to the recipient directly to build transactions over a TOR hidden service similar to how http(s) transaction building works today.

-If the recipient is not online, or cannot be contacted directly, users have the option of falling back to an asynchronous communication system where slates are encrypted and stored, for a limited period of time, by “relay nodes” (described in 0000-asynchronous-transacting-via-relays). +This RFC does not cover the offline transacting problem. # Motivation Grin is unique in that it requires the sender and receiver to interact in order to transact. This presents a lot of unique challenges that most coins don’t have to deal with. There are a number of different incompatible standards for sending and receiving, resulting in confusion and headaches for many users. The hope is that the addressing mechanism described here will become the new default method for sending and receiving, deprecating several less secure and less private methods in the process. @@ -17,17 +17,16 @@ From an end-user perspective, there should no longer be a need to configure rout ## Reference-level explanation ### Addressing -Onion addresses for TOR hidden services are generated from an ed25519 public key (32 bytes), and include a checksum and a version[1]. This provides an equivalent level of security as bitcoin addresses, and can be ephemeral or permanent, depending on the user’s needs. Grin addresses should be generated in the same way as TOR addresses, although we have the flexibility to decide whether we would like to encode them using base32 (like TOR), base58 (like Bitcoin), or even a far superior encoding based on emojis.

-Although ed25519 is a different curve than used by the grin protocol, we can still use our HD wallets to generate deterministic ed25519 public keys (and therefore Grin addresses). We can decide to generate them from a special keychain path, or even using a different master seed [2], along with a KDF for converting those HD child keys to ed25519-compatible private keys. +Onion addresses for TOR hidden services are generated from an ed25519 public key (32 bytes), and include a checksum and a version[1]. This provides an equivalent level of security as bitcoin addresses, and can be ephemeral or permanent, depending on the user’s needs. Grin addresses should be generated and base32 encoded in the same way as TOR v3 addresses.

+Although ed25519 is a different curve than used by the grin protocol, we can still use our HD wallets to generate deterministic ed25519 public keys (and therefore Grin addresses). For account `m/0`, addresses will be generated using keychain paths `m/0/1/x`, for account `m/1`, addresses will be generated using `m/1/1/x`, etc. To generate addresses for a keychain path, we derive the private key in the usual way, but then blake2b hash the derived key to get the ed25519 secret key, which can then be used to calculate the public key and address. ### TOR Hidden Services TOR hidden services can be used to directly serve the existing foreign APIs. When configuring TOR (whether bundled with grin, or installed separately), you would just publish a hidden service and configure TOR to forward all traffic to port 3420. This means we can continue supporting http(s) sending/receiving with no disruption, though it’s advisable to avoid sending directly over http(s) asap.In future versions of Grin, we can stop allowing non-local connections to the foreign wallet APIs. # Drawbacks -Requires users to setup and configure TOR, or bundle it with Grin, which could be non-trivial, and could conflict with locally installed/running versions. +Requires users to setup TOR, or bundle it with Grin, which could be non-trivial, and could conflict with locally installed/running versions. # Unresolved questions -* How should we encode addresses? base32, base58, emoji, etc. * How often should addresses change? Should users manually request a new address, or should they auto expire? Should we support multiple? # Future possibilities From 1ba2596d831ba82bdd9ff68a90a0d8221a14b3ad Mon Sep 17 00:00:00 2001 From: Antioch Peverell Date: Mon, 2 Dec 2019 13:46:27 +0000 Subject: [PATCH 35/61] Add 0006-payment-proofs to list of accepted RFCs --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 25b333d..bc80bf7 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,7 @@ To begin writing your own RFC or to find out more about the process and the gene | [0002-grin-governance](text/0002-grin-governance.md) | Articulate community values, define core and sub-teams | | [0003-security-process](text/0003-security-process.md) | Define community standards for ethical disclosure behaviour | | [0004-full-wallet-lifecycle](text/0004-full-wallet-lifecycle.md) | Define API standard for sensitive wallet operations | -| [0005-variable-size-kernels](text/0005-variable-size-kernels.md) | Introduce kernel variants that can be of different sizes | +| [0005-variable-size-kernels](text/0005-variable-size-kernels.md) | Introduce kernel variants that can be of different sizes | [0006-payment-proofs](text/0006-payment-proofs.md) | Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. ## License From 08b6a17aeee1ad8b9cc43348e861f1e2fd0977fc Mon Sep 17 00:00:00 2001 From: Antioch Peverell Date: Mon, 2 Dec 2019 13:47:25 +0000 Subject: [PATCH 36/61] fix accepted RFCs table formatting --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index bc80bf7..66ec46c 100644 --- a/README.md +++ b/README.md @@ -17,7 +17,8 @@ To begin writing your own RFC or to find out more about the process and the gene | [0002-grin-governance](text/0002-grin-governance.md) | Articulate community values, define core and sub-teams | | [0003-security-process](text/0003-security-process.md) | Define community standards for ethical disclosure behaviour | | [0004-full-wallet-lifecycle](text/0004-full-wallet-lifecycle.md) | Define API standard for sensitive wallet operations | -| [0005-variable-size-kernels](text/0005-variable-size-kernels.md) | Introduce kernel variants that can be of different sizes | [0006-payment-proofs](text/0006-payment-proofs.md) | Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. +| [0005-variable-size-kernels](text/0005-variable-size-kernels.md) | Introduce kernel variants that can be of different sizes | +| [0006-payment-proofs](text/0006-payment-proofs.md) | Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. ## License From 7d25a3668a7d30a421d587f2297aa471f9dd122b Mon Sep 17 00:00:00 2001 From: David Burkett Date: Mon, 2 Dec 2019 10:04:07 -0500 Subject: [PATCH 37/61] Update and rename 0000-payment-proofs.md to 0006-payment-proofs.md --- text/{0000-payment-proofs.md => 0006-payment-proofs.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename text/{0000-payment-proofs.md => 0006-payment-proofs.md} (96%) diff --git a/text/0000-payment-proofs.md b/text/0006-payment-proofs.md similarity index 96% rename from text/0000-payment-proofs.md rename to text/0006-payment-proofs.md index 44b270a..3438be7 100644 --- a/text/0000-payment-proofs.md +++ b/text/0006-payment-proofs.md @@ -1,8 +1,8 @@ - Title: payment-proofs - Authors: [David Burkett](mailto:davidburkett38@gmail.com) - Start date: Nov 5, 2019 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) -- Tracking issue: [Edit if merged with link to tracking github issue] +- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0006](https://github.com/mimblewimble/grin-rfcs/pull/31) +- Tracking issue: [mimblewimble/grin-wallet#265](https://github.com/mimblewimble/grin-wallet/issues/265) --- # Summary From 4b22e032826d56cbf5d0fb777d2808b0b81a03c3 Mon Sep 17 00:00:00 2001 From: Antioch Peverell Date: Tue, 3 Dec 2019 09:31:26 +0000 Subject: [PATCH 38/61] deleting file to resolve bad PR merge... --- text/0006-payment-proofs.md | 111 ------------------------------------ 1 file changed, 111 deletions(-) delete mode 100644 text/0006-payment-proofs.md diff --git a/text/0006-payment-proofs.md b/text/0006-payment-proofs.md deleted file mode 100644 index 63db37f..0000000 --- a/text/0006-payment-proofs.md +++ /dev/null @@ -1,111 +0,0 @@ -- Title: payment-proofs -- Authors: [David Burkett](mailto:davidburkett38@gmail.com) -- Start date: Nov 5, 2019 -- RFC PR: https://github.com/mimblewimble/grin-rfcs/pull/31 -- Tracking issue: https://github.com/mimblewimble/grin-wallet/issues/265 ---- - -# Summary -[summary]: #summary - -Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. - -# Motivation -[motivation]: #motivation - -Bitcoin and other cryptocurrencies with transparent protocol-level addressing and immutable, unprunable blockchains can prove sender, receiver, and amounts of payments simply by pointing to the transaction in the blockchain. -Grin's privacy and scalability means users no longer have this ability. This prevents some merchants from accepting Grin due to the high possibility of payment disputes that are unresolvable in the same way they are for transparent coins. - -This RFC changes the transaction building process where payers can require payees to create a "proof" they've received a payment before the payer finalizes and broadcasts the transaction. - -# Community-level explanation -[community-level-explanation]: #community-level-explanation - -From an end-user perspective, payers can require payees to prove receipt of funds as part of the transacting process. -Payers can then use these "proofs" to resolve payment disputes and prove they sent funds to the correct payee. - -# Reference-level explanation -[reference-level-explanation]: #reference-level-explanation - -## Slate changes - -A new (optional) structure (`payment_info`) will be added to transaction slates, along with a version increase. The `payment_info` structure will contain: - -* `sender_address` - An ed25519 public key generated by the sender. -* `receiver_address` - An ed25519 public key for the receiver, typically the public key of the user's v3 onion address. -* `receiver_signature` - A signature of the sender_address, received amount, and kernel commitment that validates against the `receiver_address`. - -## Generating proofs - -Receipt confirmations (`receiver_signature`) will be generated by the payee by providing an ED25519 signature of: `(amount || kernel_commitment || sender_address)`, using the private key of the `receiver_address`. - -The `sender_signature` can be generated for `(amount || kernel_commitment || sender_address)` using the private key of the `sender_address`. - -Sender will then create and store the following info, which can be considered the complete `payment_proof`: - -* `receiver_address` -* `receiver_signature` -* `amount` -* `kernel_commitment` -* `sender_address` -* `sender_signature` - -## Verifying Proofs - -This `payment_proof` can be provided by the sender at any time to convince a payee that a payment was made to them. The proof can be verified as follows: - -1. Ensure the `kernel_commitment` is confirmed on-chain. -2. Verify that the `receiver_address` belongs to the payee. -3. Verify that the `receiver_signature` is valid. -4. Verify that the `sender_signature` is valid. - -## Wallet actions -### init-send - -As part of the first step of the tx-building process, the sender/payer generates the `sender_address` using their keychain. - -The `receiver_address` and keychain path of the `sender_address` must be stored locally, along with the `slate_id`. - -The `sender_address` and `receiver_address` will then be added to the `payment_proof` structure of the slate. - -### receive - -If the `payment_proof` structure exists on the slate, it is mandatory that the `receiver_signature` is generated and added to the slate as part of the receive tx-building step. - -### finalize - -Using the `slate_id`, the sender can retrieve the original `sender_address` and `receiver_address` that were included in the slate, and verify that those fields remain unchanged. The sender must then validate the `receiver_signature`. If any of the original `payment_proof` slate fields were modified, or if the `receiver_signature` is invalid, the transaction must be rejected by the sender. - -Once the `payment_info` details have been validated, the sender can generate and store the `payment_proof` (See `Generating Proofs` above), and then finalize the transaction as normal. - -# Drawbacks -[drawbacks]: #drawbacks - -* Increases the size of tx slates. -* Possibility of privacy leakage through address reuse. - -# Rationale and alternatives -[rationale-and-alternatives]: #rationale-and-alternatives - -* This design works well with TOR tx building, yet is generic enough to work with all known transacting mechanisms. - -# Prior art -[prior-art]: #prior-art - -* Wallet713 implements payment proofs for grinbox transactions, which our design adapts and builds on to work more seemlessly with onion addresses and with transaction building methods that don't inherently rely on addresses. - -# Unresolved questions -[unresolved-questions]: #unresolved-questions - -* Can this be adapted to work for invoices? - -# Future possibilities -[future-possibilities]: #future-possibilities - -* Payment proofs could potentially be added to invoice payments in the future, but at the cost of an additional round of communication. - -# References -[references]: #references - -* Tx slate structure: https://github.com/mimblewimble/grin-wallet/blob/master/libwallet/src/slate.rs -* Beam's payment proof model: https://github.com/BeamMW/beam/blob/c9beb0eae55fa6b7fb3084ebe9b5db2850cf83b9/wallet/wallet_db.cpp#L3231-L3236 From 17533cc131b4690dcec7c66572dcdcdf735b7daf Mon Sep 17 00:00:00 2001 From: Antioch Peverell Date: Tue, 3 Dec 2019 10:41:28 +0000 Subject: [PATCH 39/61] cleanup RFC PR link --- text/0006-payment-proofs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0006-payment-proofs.md b/text/0006-payment-proofs.md index 3438be7..5b57296 100644 --- a/text/0006-payment-proofs.md +++ b/text/0006-payment-proofs.md @@ -1,7 +1,7 @@ - Title: payment-proofs - Authors: [David Burkett](mailto:davidburkett38@gmail.com) - Start date: Nov 5, 2019 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0006](https://github.com/mimblewimble/grin-rfcs/pull/31) +- RFC PR: [mimblewimble/grin-rfcs#0006](https://github.com/mimblewimble/grin-rfcs/pull/31) - Tracking issue: [mimblewimble/grin-wallet#265](https://github.com/mimblewimble/grin-wallet/issues/265) --- From d9832641111595b8d2a30df87dada5dd040cad4a Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Tue, 3 Dec 2019 17:04:06 +0100 Subject: [PATCH 40/61] Fix typo in grin dir --- text/0000-node-api-v2.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-node-api-v2.md b/text/0000-node-api-v2.md index b7344f3..fbb26b9 100644 --- a/text/0000-node-api-v2.md +++ b/text/0000-node-api-v2.md @@ -1079,8 +1079,8 @@ or a block that doesn't exist: ## Authentication Like the v1 API, the v2 API uses basic auth. However, the foreign and owner API do not share the same secret. -- The Owner API use the same token as the V1 Rest API, usually defined in `grin/main/.api_secret`. -- The Foreign API use its own token, usually defined in `grin/main/.foreign_api_secret`. +- The Owner API use the same token as the V1 Rest API, usually defined in `.grin/main/.api_secret`. +- The Foreign API use its own token, usually defined in `.grin/main/.foreign_api_secret`. ## Wallet support From c32985bcec60c54554a9c7c5bc3d1bd01a331a5e Mon Sep 17 00:00:00 2001 From: lehnberg Date: Thu, 5 Dec 2019 14:09:07 +0000 Subject: [PATCH 41/61] fix: Change tracking issue as per https://github.com/mimblewimble/grin-wallet/issues/265#issuecomment-561623392 --- text/0006-payment-proofs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0006-payment-proofs.md b/text/0006-payment-proofs.md index 5b57296..11cc398 100644 --- a/text/0006-payment-proofs.md +++ b/text/0006-payment-proofs.md @@ -2,7 +2,7 @@ - Authors: [David Burkett](mailto:davidburkett38@gmail.com) - Start date: Nov 5, 2019 - RFC PR: [mimblewimble/grin-rfcs#0006](https://github.com/mimblewimble/grin-rfcs/pull/31) -- Tracking issue: [mimblewimble/grin-wallet#265](https://github.com/mimblewimble/grin-wallet/issues/265) +- Tracking issue: [mimblewimble/grin-wallet#230](https://github.com/mimblewimble/grin-wallet/issues/230) --- # Summary From 405c6243746a76a38735a84644ddb095156e519a Mon Sep 17 00:00:00 2001 From: lehnberg Date: Thu, 5 Dec 2019 14:12:00 +0000 Subject: [PATCH 42/61] fix: Correct PR number This PR corrects the referenced PR number. (Sorry for the duplicate PRs, just caught the tiny bug) --- text/0006-payment-proofs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0006-payment-proofs.md b/text/0006-payment-proofs.md index 11cc398..9a16d77 100644 --- a/text/0006-payment-proofs.md +++ b/text/0006-payment-proofs.md @@ -1,7 +1,7 @@ - Title: payment-proofs - Authors: [David Burkett](mailto:davidburkett38@gmail.com) - Start date: Nov 5, 2019 -- RFC PR: [mimblewimble/grin-rfcs#0006](https://github.com/mimblewimble/grin-rfcs/pull/31) +- RFC PR: [mimblewimble/grin-rfcs#31](https://github.com/mimblewimble/grin-rfcs/pull/31) - Tracking issue: [mimblewimble/grin-wallet#230](https://github.com/mimblewimble/grin-wallet/issues/230) --- From f3946783fab95050a9e510f1d9c1f0ebf75d59bd Mon Sep 17 00:00:00 2001 From: lehnberg Date: Thu, 5 Dec 2019 14:26:28 +0000 Subject: [PATCH 43/61] RFC#0007 --- text/{0000-node-api-v2.md => 0007-node-api-v2.md} | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) rename text/{0000-node-api-v2.md => 0007-node-api-v2.md} (99%) diff --git a/text/0000-node-api-v2.md b/text/0007-node-api-v2.md similarity index 99% rename from text/0000-node-api-v2.md rename to text/0007-node-api-v2.md index fbb26b9..c956e7e 100644 --- a/text/0000-node-api-v2.md +++ b/text/0007-node-api-v2.md @@ -1,9 +1,9 @@ -- Title: Node API V2 +- Title: `node-api-v2` - Authors: [Quentin Le Sceller](mailto:q.lesceller@gmail.com) - Start date: September 30th, 2019 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) -- Tracking issue: [Edit if merged with link to tracking github issue] +- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#28](https://github.com/mimblewimble/grin-rfcs/pull/28) +- Tracking issue: [mimblewimble/grin#3158](https://github.com/mimblewimble/grin/issues/3158) --- From d5733e96ec84d083a1a446a2dc99d4bb49e5975c Mon Sep 17 00:00:00 2001 From: lehnberg Date: Thu, 5 Dec 2019 14:31:33 +0000 Subject: [PATCH 44/61] Update README.md Update README to include RFC 0007 --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 66ec46c..962bafa 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ To begin writing your own RFC or to find out more about the process and the gene | [0004-full-wallet-lifecycle](text/0004-full-wallet-lifecycle.md) | Define API standard for sensitive wallet operations | | [0005-variable-size-kernels](text/0005-variable-size-kernels.md) | Introduce kernel variants that can be of different sizes | | [0006-payment-proofs](text/0006-payment-proofs.md) | Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. +| [0007-node-api-v2](text/0007-node-api-v2.md) | Create a v2 JSON-RPC API for the Node API. ## License From 3fc9719ad4c99fc9419a08cbff1decb68249300c Mon Sep 17 00:00:00 2001 From: Yeastplume Date: Thu, 5 Dec 2019 14:33:02 +0000 Subject: [PATCH 45/61] Improved Wallet State Managment (#30) * add wallet update process rfc * update to tenses, title * typo updates, more explanation of rationale behind 'smallest' strategy * spelling errors, remove ttl by consensus question * update rfc number --- text/0007-wallet-state-management.md | 169 +++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) create mode 100644 text/0007-wallet-state-management.md diff --git a/text/0007-wallet-state-management.md b/text/0007-wallet-state-management.md new file mode 100644 index 0000000..c39714e --- /dev/null +++ b/text/0007-wallet-state-management.md @@ -0,0 +1,169 @@ + +- Title: improved-wallet-state-management +- Authors: [Michael Cordner](mailto:yeastplume@protonmail.com) +- Start date: Nov 4th, 2019 +- RFC PR: [mimblewimble/grin-rfcs#0007](https://github.com/mimblewimble/grin-rfcs/pull/30) +- Tracking issue: [mimblewimble/grin-wallet#244](https://github.com/mimblewimble/grin-wallet/issues/244) + +--- + +# Summary +[summary]: #summary + +The changes outlined in the RFC are intended to make process of updating a wallet's state from the chain more consistent for developers and more transparent to the end user. + +This includes the following changes: + +* The wallet's update process is modified to be more consistent and encapsulated. +* The `check-repair` process is run periodically as part of normal wallet update operations +* The ability for the wallet to run the update process on a separate thread is added. +* A TTL (Time-to-Live) field is added to the slate +* The default output selection method is set to `smallest` + +# Motivation +[motivation]: #motivation + +Grin wallet previously updated the state of its outputs and transactions using a combination of UTXO updates, Kernel Lookups and UTXO set scanning. These 'primitives' on their own are generally enough to keep wallet states consistent, however the manner in which they were previously invoked was less than ideal and relied heavily on manually invoking the `check-repair` process and `cancel` command. + +To rectify this situation, this RFC outlines enhancements made to the wallet update process that have the goal of ensuring wallets are always in a consistent state in a manner that is transparent to the user. After adopting these changes, most wallets should automatically keep themselves in sync with the chain in all cases, and users will not usually have to invoke the `check-repair` or `cancel` commands. + +# Community-level explanation +[community-level-explanation]: #community-level-explanation + +While these changes should be transparent to end users, wallet developers should note the changes to how the wallet updates itself via these processes. Special attention should be paid to the new update thread API functions, as well as the new TTL field and default selection strategy change. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +## Wallet Update Procedures + +The following sections outlines the processes used in the creation and update of outputs and related transactions within the wallet. + +### Transaction and Output Creation + +#### Coinbase Output + Transaction Creation + +Coinbase outputs and their related transactions are created via the wallet listener as part of the mining process. The workflow is: + + * When mining, create a potential coinbase output for the target block. + * If the block is accepted and the output is detected in the UTXO set (via the [Update by Output process](#update-by-output-process), create a transaction log entry of type 'Confirmed Coinbase', with 'confirmed' set to true. + +#### Transacting (Payer -> Payee) Output + Transaction Creation + +Outputs and tranasctions are created during the transaction exchange process as follows. Note that the invoicing workflow is mostly identical with the roles of Payer and Payee reversed, so this workflow is not outlined separately. + + * The sender creates a new 'blank' transaction Slate, adding inputs and change outputs to the slate. + * The sender generates a random kernel offset. (The kernel offset is always generated by the slate creator) + * The sender sends the slate to the payee (via file or http). If sending synchronously, (e.g. via http) the associated transaction is saved to the log after a response from the payee's listener. If sending asynchronously, (e.g. via file), the transaction is saved immediately. + * When saved, the associated transaction is set to type 'Sent Tx' with a status of 'unconfirmed', inputs are locked internally and a change output is added with status 'unconfirmed'. + * The payee receives the slate, creates (an) output(s) for the received amount with status unconfirmed, and immediately stores a transaction in their log of type 'Received Tx' with confirmed set to false. + * The payee calculates and saves the transaction kernel commitment for later reference. + * The slate is returned to the payer for completion, who calculates and saves the transaction kernel. The transaction is sent to a node for validation. + +### Transaction and Output Statue Update Processes + +#### Update by Output Process + +The Update by Output Process is the main method by which the wallet updates its internal state, decides whether a transaction has been confirmed and when to remove or confirm individual outputs. This process works as follows: + +* For every unspent output in the wallet: +* If the output's status in the user's wallet is 'unconfirmed' and it is present in the UTXO set, change the output's status to 'confirmed', and update the associated transaction status to 'confirmed'. +* If the output's status in the user's wallet is 'confirmed' and it no longer appears in the UTXO set, set its status to 'Spent'. Note these outputs will usually be locked so they cannot be selected for spending again. + +#### Kernel Lookup Process + +It is also possible to look up a transaction via the kernel that was stored when the transaction and outputs were created. This is necessary in cases such as where a participant in a transaction doesn't have any change outputs, which means the output update process won't detect when to mark a transaction confirmed. This process is: + +* Retrieve the transaction kernel from the node using the kernel excess value calculated during the transaction creation process. +* If the kernel exists, update the status of the associated transaction to 'Confirmed' + +#### Check-Repair Process + +The Update by Output process on its own is not enough to ensure the contents of a wallet is correct. There are many easily-encountered situations in the course of a wallet's operation where this process is insufficient and can potentially lead to an wallet state that's inconsistent with the UTXO set, including but not limited to: + +* Manually cancelling transactions and unlocking outputs before they've had a chance to confirm on the chain. +* Running multiple wallets from the same seed +* Fork situations + +The check-repair process fixes most of these potential issues by scanning every single output in the UTXO set and testing for ownership by determining if its bullet proof can be decoded using the wallet's master key. It will then 'check and repair' all outputs and transactions in the wallet as follows: + +* If an output exists in the UTXO set and is not in the wallet, create an output for the wallet at the derivation path stored in the bullet proof, and create a new transaction crediting the amount +* If an output exists in the UTXO set and is marked as 'spent', mark it as unspent and cancel any associated transaction log entries. + +Additionally, the check-repair process can take a flag instructing it to delete unconfirmed outputs and reset all outstanding transactions. If this flag is set: + +* If a locked output exists in the UTXO set, unlock it and set any associated transactions to `cancelled` +* If an 'unconfirmed' output is not in the UTXO set, delete it and cancel any associated transaction log entries. + +Note that the wallet `restore` process works very similarly to the check-repair process, however it must always operate on an empty wallet meaning it is limited to creating outputs and transactions in the wallet. + +## Previous Overall Update Process + +All of the procedures outlined above can be considered set of 'primitives' available for the wallet to keep itself updated. The previous update process was somewhat limited, and had the tendency to require a significant amount of manual updating using the `check-repair` command. The previous process was as follows: + +* Invoke the 'Update by Output' process before retrieving any info relating to the wallet state (`txs`, `info` or `outputs` commands,) or creating a new transaction. This invocation is usually done internally and in a syncronous blocking manner by each command that requires the wallet state to be as up to date as possible. +* During the `txs` retrieval command, for each unconfirmed transaction with an `amount recieved` field set to 0, invoke the 'Kernel Lookup' process. +* If the user suspects the wallet state to be inconsistent with the node's UTXO state, the user can run the manual `check-repair` command. This will scan all UTXOs in the set from position 1 in the output PMMR. The user can optionally provide a flag to remove all unconfirmed transactions. +* If a transaction and associated outputs appear 'stuck' or 'locked' due to the other party not completing a transaction, the user must manually cancel the transactions and unlock the outputs for re-use. (or provide the `cancel-unconfirmed` flag to the check-repair process) +* The default selection strategy is to 'sweep' the wallet of outputs on each transaction creation, meaning that by default, no further transaction can usually be made while a transaction is outstanding. + +## Changes to Overall Update Process and Procedures + +While a combination of the 'primitives' listed above should be enough to keep a wallet's state consistent with a node's UTXO set, the previous method of invoking the vital `check-repair` command was a manual step that had to be run on the entire UTXO set each time. The overall update process is modified to incorporate the previous `check-repair` logic as part of normal operation. To ensure a wallet is only scanning the part of the UTXO set required, the wallet stores details on what parts of the UTXO set it has already scanned, and performs incremental scans as part of normal update operation. + +### `check-repair` is renamed to `scan` + +The naming of the previous `check-repair` command and functionality implied something had gone wrong, whereas it really should be considered a necessary part of normal wallet operation. It is therefore renamed to a more friendly-sounding `scan` process. + +### Overall Update Process + +The Overall Update process is outlined as follows: + +* Perform the 'Update by Output' process. +* Perform the 'Kernel Update' process for transactions where the incoming amount is 0. +* Query the wallet's internal data for the last block height scanned via the `scan` process. + * If the stored block's header hash doesn't match the version currently on the node's UTXO set, scan the range of the UTXO set corresponding to the last stored block - 100 blocks up until the current block. + * If the stored block's header matches the version on the chain, scan a range of the UTXO set corresponding to the stored block height to the current block height. +* Save the last block scanned by the `scan` process. + +### New / Recovered Wallets + +Newly created and recovered wallets scan the UTXO set as follows: + +* New wallets generated with a new random seed will mark themselves as `new`, and set their `last scanned block` to the block height reported on first successful contact with a node. +* Wallets recovered with an existing mnemonic will set the `last scanned block` to `1`, triggering the `scan` process and thereby restoring any associated outputs. +* The `restore` command will be removed in favor of the `scan` process. + +### Manual Scans + +It is still be possible to manually scan the chain via the `scan` (`check`) command, however the command is modified to allow the user to specify a range of block heights to scan. If this is not provided, the scan occurs from the current block - 1 week's worth of blocks. + +### Invocation of Update Process + +The update process was previously run inline in a synchronous blocking fashion as part of individual wallet commands. With the `scan` command becoming part of normal operations, it's expected that a particular invocation of the overall update process could potentially take a long time. This is acceptable in single-use modes such as the single invocation model of `grin-wallet`, but is far less usable in environments where the wallet and Owner API stay resident and the caller may not have any particular insight as to why a call invoking the update process might be taking a long time to complete. + +To address this, the wallet provides a method of calling the Overall Update Process in its own thread that attempts to minimise its usage of the wallet lock. The Wallet's Owner API (V3 Only) is extended with the following functions: + +* `start_update_thread(interval)` => Starts the update thread, calling the Overall Update Process at the frequency specified +* `stop_update_thread()` => Stops the update thread +* `get_update_status()` => Returns the current status of the update thread, along with as much information as possible about what the thread is doing and, in the case of the `scan` process, the percentage complete. + +Further, if the update thread is currently running, commands that previously always called the Overall Update Process (such as `info`, `txs`, etc...) do not call the update process. This means that when invoked via the command line, the behaviour is unchanged. However, wallets running the Owner API can instead choose to run and monitor the background update thread at the desired frequency while keeping the user informed of any long-running update processes. + +### Transaction TTL + +The Update process outlined above does not cover the case where a slate is exchanged and outputs are locked, but one party doesn't complete the transaction for whatever reason. + +Previously, a situation such as this meant that the outputs associated with a transaction sat locked indefinitely within a user's wallet, with the only way to unlock them a manual `cancel` command (or a `check-repair` with the `delete-unconfirmed` flag set). Worse, the wallet's default selection strategy was set to `all` meaning an entire wallet's contents usually became locked on each transaction, and thus a single uncompleted transaction would render a user's entire wallet balance unspendable. + +To rectify these problems, the following changes are made: + +* A Time-To-Live (`TTL`) field is added to the Slate, which is defined as the last block height at which a wallet should attempt to complete a transaction. This field should be respected at the wallet-exchange level, but is not currently commit to at a consensus level. If a wallet detects a particular transaction's TTL has expired, it will automatically cancel the transaction and unlock associated outputs. Note this doesn't prevent the other party sending the transaction to the node after this happens, but there is no guarantee the outputs will still be available. However, if this does happen the `scan` command will correct the wallet state. +* The default output selection method is changed to `smallest`, to prevent all wallet amounts being locked on every transaction. The `smallest` strategy prefers using the smallest (or 'dust') outputs as inputs to a transaction, and it is conjectured that the overall effect on Grin's UTXO size will be negligible in the longer term, provided users continue to transact. The `all` method of output selection also has privacy implications in that it makes it easier for an observer to identify a group of outputs as belonging to a single wallet (or indeed, representing the entire contents of a wallet in most cases). + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- Do we limit the Kernel Lookup part of the process to just cases where there is no change output. Is there any benefit to calling it for all outstanding transactions on each lookup (or any particular downside to doing so?) +- What is the effect on UTXO set size of changing the selection method to `smallest`? +- Confirm sensible defaults for how far to scan back on a `scan` From fe579645262f3c3ed826c72186803dffe86bcc1b Mon Sep 17 00:00:00 2001 From: Quentin Le Sceller Date: Thu, 5 Dec 2019 15:42:25 +0100 Subject: [PATCH 46/61] Cleanup numbering, add to README and remove 'Edit if merged' (#35) --- README.md | 1 + text/0003-security-process.md | 2 +- text/0007-node-api-v2.md | 2 +- ...llet-state-management.md => 0008-wallet-state-management.md} | 2 +- 4 files changed, 4 insertions(+), 3 deletions(-) rename text/{0007-wallet-state-management.md => 0008-wallet-state-management.md} (99%) diff --git a/README.md b/README.md index 962bafa..ee91e26 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,7 @@ To begin writing your own RFC or to find out more about the process and the gene | [0005-variable-size-kernels](text/0005-variable-size-kernels.md) | Introduce kernel variants that can be of different sizes | | [0006-payment-proofs](text/0006-payment-proofs.md) | Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. | [0007-node-api-v2](text/0007-node-api-v2.md) | Create a v2 JSON-RPC API for the Node API. +| [0008-wallet-state-management](text/0008-wallet-state-management.md) | Improve wallet state management. ## License diff --git a/text/0003-security-process.md b/text/0003-security-process.md index 5101d7d..1b030d5 100644 --- a/text/0003-security-process.md +++ b/text/0003-security-process.md @@ -2,7 +2,7 @@ - Title: security-process - Authors: [John Woeltz](mailto:joltz@protonmail.com) - Start date : July 18, 2019 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#13](https://github.com/mimblewimble/grin-rfcs/pull/13) +- RFC PR: [mimblewimble/grin-rfcs#13](https://github.com/mimblewimble/grin-rfcs/pull/13) - Tracking issue: [mimblewimble/grin-pm#178](https://github.com/mimblewimble/grin-pm/issues/178) --- diff --git a/text/0007-node-api-v2.md b/text/0007-node-api-v2.md index c956e7e..e8d084f 100644 --- a/text/0007-node-api-v2.md +++ b/text/0007-node-api-v2.md @@ -2,7 +2,7 @@ - Title: `node-api-v2` - Authors: [Quentin Le Sceller](mailto:q.lesceller@gmail.com) - Start date: September 30th, 2019 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#28](https://github.com/mimblewimble/grin-rfcs/pull/28) +- RFC PR: [mimblewimble/grin-rfcs#28](https://github.com/mimblewimble/grin-rfcs/pull/28) - Tracking issue: [mimblewimble/grin#3158](https://github.com/mimblewimble/grin/issues/3158) --- diff --git a/text/0007-wallet-state-management.md b/text/0008-wallet-state-management.md similarity index 99% rename from text/0007-wallet-state-management.md rename to text/0008-wallet-state-management.md index c39714e..f6b2a45 100644 --- a/text/0007-wallet-state-management.md +++ b/text/0008-wallet-state-management.md @@ -2,7 +2,7 @@ - Title: improved-wallet-state-management - Authors: [Michael Cordner](mailto:yeastplume@protonmail.com) - Start date: Nov 4th, 2019 -- RFC PR: [mimblewimble/grin-rfcs#0007](https://github.com/mimblewimble/grin-rfcs/pull/30) +- RFC PR: [mimblewimble/grin-rfcs#30](https://github.com/mimblewimble/grin-rfcs/pull/30) - Tracking issue: [mimblewimble/grin-wallet#244](https://github.com/mimblewimble/grin-wallet/issues/244) --- From 9780c82fc67b1f9680f5d51479a531f1f14d5179 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Wed, 18 Dec 2019 05:39:46 -0500 Subject: [PATCH 47/61] Updating PR and Tracking Issue --- text/0000-enable-faster-sync.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/text/0000-enable-faster-sync.md b/text/0000-enable-faster-sync.md index 6b8433e..0698e98 100644 --- a/text/0000-enable-faster-sync.md +++ b/text/0000-enable-faster-sync.md @@ -2,8 +2,8 @@ - Title: enable-faster-sync - Authors: [Antioch Peverell](mailto:apeverell@protonmail.com), [John Tromp](mailto:john.tromp@gmail.com) - Start date: Oct 25, 2019 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) -- Tracking issue: [Edit if merged with link to tracking github issue] +- RFC PR: [mimblewimble/grin-rfcs#29](https://github.com/mimblewimble/grin-rfcs/pull/29) +- Tracking issue: [mimblewimble/grin-wallet#3173](https://github.com/mimblewimble/grin-wallet/issues/3173) --- From bbc247b95ef219da23c3fa5453103b21e6bfbb54 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Wed, 18 Dec 2019 05:40:29 -0500 Subject: [PATCH 48/61] Rename 0000-enable-faster-sync.md to 0009-enable-faster-sync.md --- text/{0000-enable-faster-sync.md => 0009-enable-faster-sync.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-enable-faster-sync.md => 0009-enable-faster-sync.md} (100%) diff --git a/text/0000-enable-faster-sync.md b/text/0009-enable-faster-sync.md similarity index 100% rename from text/0000-enable-faster-sync.md rename to text/0009-enable-faster-sync.md From 6658fde64ebb5ca923c8ae9b69ffb65364dc00b8 Mon Sep 17 00:00:00 2001 From: David Burkett Date: Wed, 18 Dec 2019 05:44:50 -0500 Subject: [PATCH 49/61] Adding 0009-enable-faster-sync --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index ee91e26..13ddc5d 100644 --- a/README.md +++ b/README.md @@ -21,6 +21,7 @@ To begin writing your own RFC or to find out more about the process and the gene | [0006-payment-proofs](text/0006-payment-proofs.md) | Support generating and validating payment proofs for sender-initiated (i.e. non-invoice) transactions. | [0007-node-api-v2](text/0007-node-api-v2.md) | Create a v2 JSON-RPC API for the Node API. | [0008-wallet-state-management](text/0008-wallet-state-management.md) | Improve wallet state management. +| [0009-enable-faster-sync](text/0009-enable-faster-sync.md) | Enable faster txhashset sync by changing output MMR commitment ## License From c347843c81148273946581c7079183b845d4eaee Mon Sep 17 00:00:00 2001 From: Yeastplume Date: Wed, 18 Dec 2019 15:47:08 +0000 Subject: [PATCH 50/61] merge tor rfc --- README.md | 1 + text/0000-wallet-update-process.md | 168 ------------------ ....md => 0010-online-transacting-via-tor.md} | 12 +- 3 files changed, 9 insertions(+), 172 deletions(-) delete mode 100644 text/0000-wallet-update-process.md rename text/{0000-online-transacting-via-tor.md => 0010-online-transacting-via-tor.md} (91%) diff --git a/README.md b/README.md index 13ddc5d..214b33f 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,7 @@ To begin writing your own RFC or to find out more about the process and the gene | [0007-node-api-v2](text/0007-node-api-v2.md) | Create a v2 JSON-RPC API for the Node API. | [0008-wallet-state-management](text/0008-wallet-state-management.md) | Improve wallet state management. | [0009-enable-faster-sync](text/0009-enable-faster-sync.md) | Enable faster txhashset sync by changing output MMR commitment +| [0010-online-transacting-via-tor](text/0010-online-transacting-via-tor.md) | Define standard for transacting via TOR ## License diff --git a/text/0000-wallet-update-process.md b/text/0000-wallet-update-process.md deleted file mode 100644 index b2e680e..0000000 --- a/text/0000-wallet-update-process.md +++ /dev/null @@ -1,168 +0,0 @@ - -- Title: wallet-update-process -- Authors: [Michael Cordner](mailto:yeastplume@protonmail.com) -- Start date: Nov 4th, 2019 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) -- Tracking issue: [mimblewimble/grin-wallet#244](https://github.com/mimblewimble/grin-wallet/issues/244) - ---- - -# Summary -[summary]: #summary - -The changes outlined in the RFC are intended to make process of updating a wallet's state from the chain more consistent for developers and more transparent to the end user. The following changes are proposed: - -* Modify the wallet's update process to be more consistent and encapsulated. -* Run the current `check-repair` process as part of normal wallet update operations -* Provide the ability to run the wallet update process on a separate thread. -* Incorporate a TTL (Time-to-Live) field in the Slate -* Change the default output selection method to `smallest` - -# Motivation -[motivation]: #motivation - -The grin wallet currently updates the state of its outputs and transactions using a combination of UTXO updates, Kernel Lookups and UTXO set scanning. These 'primitives' on their own are generally enough to keep wallet states consistent, however the manner in which they are currently invoked is less than ideal and relies heavily on manually invoking the `check-repair` process and `cancel` command. - -To rectify this situation, this RFC outlines changes to the wallet update process that have the goal of ensuring wallets are always in a consistent state in a manner that is transparent to the user. It is hoped that upon adopting these changes, most wallets will automatically keep themselves in sync with the chain in all cases, and that users will not usually have to invoke the `check-repair` or `cancel` commands. - -# Community-level explanation -[community-level-explanation]: #community-level-explanation - -While these changes should be transparent to end users, wallet developers should note the changes to how the wallet updates itself via these processes. Special attention should be paid to the new update thread API functions, as well as the new TTL field and default selection strategy change. - -# Reference-level explanation -[reference-level-explanation]: #reference-level-explanation - -## Current Wallet Update Procedures - -The following sections outline the current processes used in the creation and update of outputs and related transactions within the wallet. - -### Transaction and Output Creation - -#### Coinbase Output + Transaction Creation - -Coinbase outputs and their related transactions are created via the wallet listener as part of the mining process. The currrent workflow is: - - * When mining, create a potential coinbase output for the target block. - * If the block is accepted and the output is detected in the UTXO set (via the [Update by Output process](#)), create a transaction log entry of type 'Confirmed Coinbase', with 'confirmed' set to true. - -#### Transacting (Payer -> Payee) Output + Transaction Creation - -Outputs and tranasctions are created during the transaction exchange process as follows. Note that the invoicing workflow is mostly identical with the roles of Payer and Payee reversed, so this workflow is not outlined separately. - - * The sender creates a new 'blank' transaction Slate, adding inputs and change outputs to the slate. - * The sender generates a random kernel offset. (The kernel offset is always generated by the slate creator) - * The sender sends the slate to the payee (via file or http). If sending synchronously, (e.g via http) the associated transaction is saved to the log after a response from the payee's listener. If sending asynchronously, (e.g. via file), the transaction is saved immediately. - * When saved, the associated transaction is set to type 'Sent Tx' with a status of 'unconfirmed', inputs are locked internally and a change output is added with status 'unconfirmed'. - * The payee receives the slate, creates (an) output(s) for the received amount with status unconfirmed, and immediately stores a transaction in their log of type 'Received Tx' with confirmed set to false. - * The payee calculates and saves the transaction kernel for later reference. - * The slate is returned to the payer for completion, who calculates and saves the transaction kernel. The transaction is sent to a node for validation. - -### Transaction and Output Statue Update Processes - -#### Update by Output Process - -The Update by Output Process has been the main method by which the wallet updates its internal state, deciding whether a transaction has been confirmed and when to remove or confirm individual outputs. This process works as follows: - -* For every unspent output in the wallet: -* If the output's status in the user's wallet is 'unconfirmed' and it is present in the UTXO set, change the output's status to 'confirmed', and update the associated transaction status to 'confirmed'. -* If the output's status in the user's wallet is 'confirmed' and it no longer appears in the UTXO set, set it's status to 'Spent'. Note these outputs will usually be locked so they cannot be selected for spending again. - -#### Kernel Lookup Process - -It is also possible to look up a transaction via the kernel that was stored when the transaction and outputs were created. This is necessary in cases such as where a participant in a transaction doesn't have any change outputs, which means the output update process won't detect when to mark a transaction confirmed. This process is: - -* Retrieve the transction kernel from the node using the kernel excess value calculated during the transaction creation process. -* If the kernel exists, update the status of the associated transaction to 'Confirmed' - -#### Check-Repair Process - -The Update by Output process on its own is not enough to ensure the contents of a wallet is correct. There are many easily-encountered situations in the course of a wallet's operation where this process is insufficient and can potentially lead to an wallet state that's inconsistent with the UTXO set, including but not limited to: - -* Manually cancelling transctions and unlocking outputs before they've had a chance to confirm on the chain. -* Running multiple wallets from the same seed -* Fork situations - -The check-repair process fixes most of these potential issues by scanning every single output in the UTXO set and testing for ownership by determining if its bullet proof can be decoded using the wallet's master key. It will then 'check and repair' all outputs and transactions in the wallet as follows: - -* If an output exists in the UTXO set and is not in the wallet, create an output for the wallet at the derivation path stored in the bullet proof, and create a new transaction crediting the amount -* If an output exists in the UTXO set and is marked as 'spent', mark it as unspent and cancel any associated transaction log entries. - -Additionally, the check-repair process can take a flag instructing it to delete unconfirmed outputs and reset all outstanding transctions. If this flag is set: - -* If a locked output exists in the UTXO set, unlock it and set any associated transactions to `cancelled` -* If an 'unconfirmed' output is not in the UTXO set, delete it and cancel any associated transction log entries. - -Note that the wallet `restore` process works very similarly to the check-repair process, however it must always operate on an empty wallet meaning it is limited to creating outputs and transactions in the wallet. - -## Current Overall Update Process - -All of the procedures outlined above can be considered set of 'primitives' available for the wallet to keep itself updated. The current update process is somewhat limited, and has the tendency to require a significant amount of manual updating using the `check-repair` command. The process today is as follows: - -* Invoke the 'Update by Output' process before retrieving any info relating to the wallet state (`txs`, `info` or `outputs` commands,) or creating a new transaction. This invocation is usually done internally and in a syncronous blocking manner by each command that requires the wallet state to be as up to date as possible. -* During the `txs` retrieval command, for each unconfirmed transaction with an `amount recieved` field set to 0, invoke the 'Kernel Lookup' process. -* If the user suspects the wallet state to be inconsistent with the node's UTXO state, the user can run the manual `check-repair` command. This will scan all UTXOs in the set from position 1 in the output PMMR. The user can optionally provide a flag to remove all unconfirmed transactions. -* If a transaction and associated outputs appear 'stuck' or 'locked' due to the other party not completing a transaction, the user must manually cancel the transactions and unlock the outputs for re-use. (or provide the `cancel-unconfirmed` flag to the check-repair process) -* The default selection strategy is to 'sweep' the wallet of outputs on each transaction creation, meaning that by default, no further transaction can usually be made while a transaction is outstanding. - -## Changes to Overall Update Process and Procedures - -While a combination of the 'primitives' listed above should be enough to keep a wallet's state consistent with a node's UTXO set, the current method of invoking the vital `check-repair` command is manual, and must be run on the entire UTXO each time. The overall update process will be modified to incorporate the current `check-repair` logic as part of normal operation. To ensure a wallet is only scanning the part of the UTXO set required, the wallet will store details on what parts of the UTXO set it has already scanned, and perform scans incrementally as part of normal update operation. - -### Rename `check-repair` to `scan` - -The naming of current `check-repair` command and functionality implies something has gone wrong, whereas it really should be considered a necessary part of normal wallet operation. It will therefore be renamed to a more friendly-sounding `scan` process. - -### New Overall Update Process - -The Overall Update process will proceed as follows: - -* Perform the 'Update by Output' process. -* Perform the 'Kernel Update' process for transactions where the incoming amount is 0. -* Query the wallet's internal data for the last block height scanned via the `scan` process. - * If the stored block's header hash doesn't match the version currently on the node's UTXO set, scan the range of the UTXO set corresponding to the last stored block - 100 blocks up until the current block. - * If the stored block's header matches the version on the chain, scan a range of the UTXO set corresponding to the stored block height to the current block height. -* Save the last block scanned by the `scan` process. - -### New / Recovered Wallets - -Newly created and recovered wallets will scan the UTXO set as follows: - -* New wallets generated with a new random seed will mark themselves as `new`, and set their `last scanned block` to the block height reported on first successful contact with a node. -* Wallets recovered with an existing mnemonic will set the `last scanned block` to `1`, triggering the `scan` process and thereby restoring any associated outputs. -* The `restore` command will be removed in favor of the `scan` process. - -### Manual Scans - -It will still be possible to manually scan the chain via the `scan` (`check`) command, however the command will be modified to allow the user to specify a range of block heights to scan. If this is not provided, the scan will occur from the current block - 1 week's worth of blocks. - -### Invocation of Update Process - -The update process is currently run inline in a synchronous blocking fashion as part of individual wallet commands. With the `scan` command becoming part of normal operations, it's expected that a particlar invocation the overall update process could potentially take a long time. This is acceptable in single-use modes such as the current single invocation model of `grin-wallet`, but will be far less usable in environments where the wallet and Owner API stay resident and the caller may not have any particular insight as to why a call invoking the update process might be taking a long time to complete. - -To address this, the wallet will provide a method of calling the Overall Update Process in its own thread that will attempt to mimimise its usage of the wallet lock. The Wallet's Owner API (V3 Only) will be extended with the following functions: - -* `start_update_thread(interval)` => Starts the update thread, calling the Overall Update Process at the frequency specified -* `stop_update_thread()` => Stops the update thread -* `get_update_status()` => Returns the current status of the update thread, along with as much information as possible about what the thread is doing and, in the case of the `scan` process, the percentage complete. - -Further, if the update thread is currently running, commands that currently call the Overall Update Process (such as `info`, `txs`) etc will not call the update process. This means that when invoked via the current command line, the behaviour should be unchanged. However, wallets running the Owner API can instead choose to run and monitor the background update thread at the desired frequency while keeping the user informed of any long-running update processes. - -### Transaction TTL - -The Update process outlined above does not cover the case where a slate is exchanged and outputs are locked, but one party doesn't complete the transaction for whatever reason. - -Currently, a situation such as this means that the outputs associated with a transaction sit locked indefinitely within a user's wallet, meaning that the only way to recover the outputs is with a manual `cancel` command (or a `check-repair` with the `delete-unconfirmed` flag set). Worse, the wallet's default selection strategy is currently set to `all` meaning an entire wallet's contents become locked on each transaction, meaning a single uncompleted transaction will render a user's wallet unusable. - -To rectify these problems, the following changes will be made: - -* The addition of a Time-To-Live (`TTL`) field to the Slate, which will be defined as the last block height at which a wallet should attempt to complete a transaction. This field should be respected at the wallet-exchange level, but is not currently commit to at a consensus level. If a wallet detects a particlar transaction's TTL has expired, it will automatically cancel the transaction and unlock associated outputs. Note this doesn't prevent the other party sending the transaction to the node after this happens, but there is no guarantee the outputs will still be available. However, if this does happen the `scan` command will correct the wallet state. -* Change the default output selection method to `smallest`, to prevent all wallet amounts being locked on every transaction. - -# Unresolved questions -[unresolved-questions]: #unresolved-questions - -- Do we limit the Kernel Lookup part of the process to just cases where there is no change output. Is there any benefit to calling it for all outstanding transactions on each lookup (or any particular downside to doing so?) -- Can we add the TTL field at consensus level? Making it consensus means much less scope for inconsistency between wallets overall, but how would it be commit to? (Could it be inclded in the kernel excess somehow, for example?) -- What is the effect on UTXO set size of changing the selection method to `smallest`? -- Confirm sensible defaults for how far to scan back on a `scan` \ No newline at end of file diff --git a/text/0000-online-transacting-via-tor.md b/text/0010-online-transacting-via-tor.md similarity index 91% rename from text/0000-online-transacting-via-tor.md rename to text/0010-online-transacting-via-tor.md index 6fce219..11f75a1 100644 --- a/text/0000-online-transacting-via-tor.md +++ b/text/0010-online-transacting-via-tor.md @@ -1,7 +1,11 @@ -# 0000-online-transacting-via-tor
-Title: online-transacting-via-tor
-Authors: DavidBurkett
-Start date: September 1, 2019
+ +- Title: online-transacting-via-tor +- Authors: [David Burkett](mailto:davidburkett38@gmail.com) +- Start date: September 1, 2019 +- RFC PR: [mimblewimble/grin-rfcs#24](https://github.com/mimblewimble/grin-rfcs/pull/24) +- Tracking issue: [mimblewimble/grin-wallet#280](https://github.com/mimblewimble/grin-wallet/issues/280) + +--- # Summary Describes a standardized addressing and communication protocol for building Grin transactions.

From ad9ca1596edea0960feb8a781c338e6b96c775bd Mon Sep 17 00:00:00 2001 From: joltz Date: Fri, 24 Jan 2020 11:14:24 -0800 Subject: [PATCH 51/61] Add security-subteam RFC draft --- text/0000-security-subteam.md | 180 ++++++++++++++++++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100644 text/0000-security-subteam.md diff --git a/text/0000-security-subteam.md b/text/0000-security-subteam.md new file mode 100644 index 0000000..d05a5ca --- /dev/null +++ b/text/0000-security-subteam.md @@ -0,0 +1,180 @@ + +- Title: security-subteam +- Authors: [joltz](mailto:joltz@protonmail.com) +- Start date: Jan 22 2020 +- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) +- Tracking issue: [Edit if merged with link to tracking github issue] + +--- + +# Summary +[summary]: #summary + +The security subteam for Grin will carry out the security priorities of the Grin community and core team, which are to be defined prior to formation of the subteam. The security subteam will streamline the security decision making process for the core team while maintaining a careful balance of security and transparency for the community. + +# Motivation +[motivation]: #motivation + +* Improve transparency, efficiency, decisiveness and scalability around security-related decision making +* Dedicated support for security-related initiatives such as bilateral disclosure agreements, bug bounty programs and audit work +* Increase confidence in the ecosystem for future contributions + +A clear and structured decision making process is needed to fully support the current security process as well as community initiatives such as bilateral disclosure agreements, bug bounty programs and future audit work. Currently this process is maintained by the core team who have many areas of focus, leading to a lack of clarity, timeliness and ability to decisively execute on community security initiatives. + +The security subteam will free up core resources to focus on other important areas while expanding dedicated resources to security work, allowing the Grin ecosystem to continue to scale. A clear decision making structure around the security process instills confidence in the greater ecosystem and encourages projects, developers and users to continue to increase contributions of time and resources into Grin. + +# Community-level explanation +[community-level-explanation]: #community-level-explanation + +Grin's security subteam is responsible for making security related decisions for the Grin project. This includes, but is not limited to, decisions about the specification and implementation of Grin's security processes. As much as possible Grin's security subteam carries out the security priorities of the community and core team in a transparent manner. + +There may be some unique security-related cases that require the security subteam to make some decisions in private to protect the security of Grin users. In these cases, once it is safe to do so, the security subteam is responsible for sharing as many details as possible with the community in the name of transparency. + +The primary day-to-day impact this will have on the community is making available structured support around security-related initiatives such as bilateral disclosure agreements, bug bounty programs and audit work. The community and other subteams will have regular opportunity to interact with, provide input to and receive input from the security subteam. + +## Example + +Here we will walk through an example case that highlights how Grin's security subteam would operate, contrasting with the current process. + +_In the example case, a vulnerability is disclosed to the disclosure contacts listed in Grin's security policy. It is shown that the community gains greater transparency into the decision making process and related outcomes with the proposed security subteam. Additionally the disclosure map is more tightly contained in the proposed process, resulting in less risk of vulnerability exploitation to the community._ + +### Current Process + +* Grin's security contacts make initial assessment of the validity of the vulnerability disclosure +* If disclosure determined to be valid, all details of the disclosure are shared with the entire core team +* With a conversation between the entire core team, a plan of action is determined and executed, following Grin's responsible disclosure policy +* Once the vulnerability is mitigated the core team may optionally share some of the details with the community + +### Proposed Process +* Grin's security contacts, all members of the security subteam, make initial assessment of the validity of the vulnerability disclosure +* If disclosure determined to be valid, a conversation and plan of action is determined and executed by the security subteam, following Grin's responsible disclosure policy +* Once the vulnerability is mitigated the security subteam must disclose all information that is not still security-critical with the community + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +Grin's security subteam will be implemented as stated below. + +## Membership + +* Bootstrapped by existing security contacts (@lehnberg, @hashmap, @joltz) +* Minimum three, maximum five members +* Membership is recommended by the security subteam and approved by the core team +* There must be at least one well-represented core developer on the subteam, chosen by the core team +* All members must have relevant domain expertise in security, cryptography, the Grin codebase or Grin community +* All members must have high competency in maintaining personal operational security (keeping private keys secure, maintaining secure communications channels etc.) +* Membership is revoked by failure to update the canary, unanimous decision by subteam or decision from the core team + * Note if the core team engages in removing members from the security subteam without reasonable justification, the security subteam loses purpose, becomes ineffective and is relegated to useless security theater + +### Core Representation + +The core team is responsible for choosing one developer who they trust to represent the core interests in the security subteam. This ensures that the core team has adequate representation in security related decisions, without overpowering the majority of discussion. Additionally, with the veto rules defined below, we ensure that in cases of disagreement the security subteam cannot make an impactful decision without the core team's awareness and input. + +## Decision Making + +* Decisions are made primarily with consideration to the security priorities outlined prior to the subteam's formation +* Unanimous approval of security subteam members is needed to make all impactful decisions +* Some decisions made must also be approved by the core team to be carried out: + * Adding/removing parties from disclosure agreements + * Adding members to the security subteam + * Updating the security policy + * If the core team does not follow a security subteam decision that it has final approval on, a reasonable explanation must be provided to the security subteam +* Some decisions must be made without fully informing the core team of all details: + * When a vulnerability requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard) + * In these cases not all information will be shared with core team, such as specific details to carry out an exploit, full impact and other relevant information that will create unnecessary risk if disclosed + * The core team must be notified of all cases of these decisions within a reasonable time frame that does not pose a security risk (e.g. after a fix is implemented) and provided with a reasonable explanation by the security subteam + +### Vetoes + +* Any member of the security subteam may veto an impactful decision + +The ability to veto an impactful decision is important to preserve the ability for members of the security subteam to make honest recommendations. Important and controversial decisions will be made and members of the security subteam need the ability to not sign their names to a decision they are not comfortable with that could have serious ramifications. These cases will be handled according to the Deadlocks section below. + +The security subteam should always strive to reach its own decisions. Constant vetoes and inability to reach unanimous decisions will dampen the advantages of the existence of the team and may render it obsolete. It is important to strike the right balance of different perspectives, expertise and reasonability in members to take advantage of the benefits that come with a diverse and functional security-related decision making process. + +### Deadlocks + +* Impactful decisions that can't be made unanimously by the security subteam will be made by the core team according to their decision making process + * If the core team is deadlocked, the decision made will be whichever decision, if any, will provide the end user with the greatest measure of security + +Deadlocks should be avoided at all costs. The inability of the security subteam to reach a unanimous decision based on the security priorities of Grin's community and core team should be a red flag to the community and core team for the health of the subteam. All members should be able to clearly discuss the factual merits of the options and reach a decision that is in the clear best possible interest for the security of Grin's ecosystem and users. + +## Example + +_By following the previous example in more detail, critical differences can be further explored between the current process and the proposed security subteam process in the handling of a unique case._ + +_In cases where the vulnerability is so critical that it requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard), the proposed process reduces the likelihood of the vulnerability being exploited with a focused decision making process and by restricting the details to essential persons only._ + +### Current Process + +* Grin's security contacts make initial assessment of the validity of the vulnerability disclosure +* It is determined that the disclosed vulnerability is a critical inflation bug +* _All details of the disclosure, including the steps to carry out the exploit, are shared with the entire core team_ +* With a conversation between the entire core team, a plan of action is determined and executed, following Grin's responsible disclosure policy + * _This conversation may take days or weeks as there is no particular decision making process for security-related issues_ +* _Once the vulnerability is mitigated the core team may optionally share some of the details with the community_ + +### Proposed Process + +* Grin's security contacts, all members of the security subteam, make initial assessment of the validity of the vulnerability disclosure +* It is determined that the disclosed vulnerability is a critical inflation bug +* _A conversation and plan of action is quickly and unanimously determined and executed on by the security subteam, following Grin's responsible disclosure policy_ +* _Once the vulnerability is mitigated, the security subteam must disclose all information that is not still security-critical with the community_ + +# Drawbacks +[drawbacks]: #drawbacks + +* The core team is already busy +* Even though it decentralizes the core team, it potentially creates another point of centralization +* Adds resource overhead to the project that _must_ be upheld (it is not acceptable to fail to follow through in security processes once established!) +* Possibly distracts from other important work- we don't want to spread attention too wide too early +* Adds bulk to Grin's governance structure +* The community may trust core more than a subteam to make critical security decisions + +# Rationale and alternatives +[rationale-and-alternatives]: #rationale-and-alternatives + +* One alternative, currently in use, is adding the input of a security expert to the existing core decision making process + * This is better than nothing but we lose a few benefits without a dedicated security subteam: + * Strengthened security process: reduced likelihood of exploit leaks + * Improved transparency: there is currently little transparency around core's security decision making process + * Improved reaction times: informed, thoughtful, decisive actions with input from relevant experts only + * Possibility for security initiatives: clear decision making process for bilateral disclosure agreements, bug bounty programs, audit work + * Stronger commitments to the community, ecosystem and other projects: viable with a dedicated, clear decision making process + * Long-term scalability: extend capacity for security work to grow beyond the core team's attention capacity + +# Prior art +[prior-art]: #prior-art + +grin-governance[0] and security-process[1] RFCs lay groundwork for the security-subteam RFC. + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- Is this the most minimal model that can have long-term success for Grin's security-related decision making process? +- Is the single-veto model sufficient for a consistent, timely decision making process? +- If core always has at least one appointed member, and one veto gives the decision to the core team, does this significantly weaken the security subteam's purpose? + +# Future possibilities +[future-possibilities]: #future-possibilities + +* Budget allocation and management + * Manage an allocated budget for possible initiatives + * Bug bounty programs + * Audits + * Full time contributors + * Development of security tooling + +* Security Subteam working groups + * Red team + * Blue team + * Audit team + * Bug bounty team + * Community outreach team + +# References +[references]: #references + +[0] https://github.com/mimblewimble/grin-rfcs/blob/master/text/0002-grin-governance.md#sub-teams + +[1] https://github.com/mimblewimble/grin-rfcs/blob/master/text/0003-security-process.md From 2f02234537a57fc876519ff71fdfdf325a0fae4b Mon Sep 17 00:00:00 2001 From: joltz Date: Tue, 28 Jan 2020 11:38:39 -0800 Subject: [PATCH 52/61] Update veto, deadlock, subteam language --- text/0000-security-subteam.md | 72 +++++++++++++++++------------------ 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/text/0000-security-subteam.md b/text/0000-security-subteam.md index d05a5ca..710eaaa 100644 --- a/text/0000-security-subteam.md +++ b/text/0000-security-subteam.md @@ -1,5 +1,5 @@ -- Title: security-subteam +- Title: security-team - Authors: [joltz](mailto:joltz@protonmail.com) - Start date: Jan 22 2020 - RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) @@ -10,7 +10,7 @@ # Summary [summary]: #summary -The security subteam for Grin will carry out the security priorities of the Grin community and core team, which are to be defined prior to formation of the subteam. The security subteam will streamline the security decision making process for the core team while maintaining a careful balance of security and transparency for the community. +The security team for Grin will carry out the security priorities of the Grin community and core team, which are to be defined prior to formation of the team. The security team will streamline the security decision making process for the core team while maintaining a careful balance of security and transparency for the community. # Motivation [motivation]: #motivation @@ -21,22 +21,22 @@ The security subteam for Grin will carry out the security priorities of the Grin A clear and structured decision making process is needed to fully support the current security process as well as community initiatives such as bilateral disclosure agreements, bug bounty programs and future audit work. Currently this process is maintained by the core team who have many areas of focus, leading to a lack of clarity, timeliness and ability to decisively execute on community security initiatives. -The security subteam will free up core resources to focus on other important areas while expanding dedicated resources to security work, allowing the Grin ecosystem to continue to scale. A clear decision making structure around the security process instills confidence in the greater ecosystem and encourages projects, developers and users to continue to increase contributions of time and resources into Grin. +The security team will free up core resources to focus on other important areas while expanding dedicated resources to security work, allowing the Grin ecosystem to continue to scale. A clear decision making structure around the security process instills confidence in the greater ecosystem and encourages projects, developers and users to continue to increase contributions of time and resources into Grin. # Community-level explanation [community-level-explanation]: #community-level-explanation -Grin's security subteam is responsible for making security related decisions for the Grin project. This includes, but is not limited to, decisions about the specification and implementation of Grin's security processes. As much as possible Grin's security subteam carries out the security priorities of the community and core team in a transparent manner. +Grin's security team is responsible for making security related decisions for the Grin project. This includes, but is not limited to, decisions about the specification and implementation of Grin's security processes. As much as possible Grin's security team carries out the security priorities of the community and core team in a transparent manner. -There may be some unique security-related cases that require the security subteam to make some decisions in private to protect the security of Grin users. In these cases, once it is safe to do so, the security subteam is responsible for sharing as many details as possible with the community in the name of transparency. +There may be some unique security-related cases that require the security team to make some decisions in private to protect the security of Grin users. In these cases, once it is safe to do so, the security team is responsible for sharing as many details as possible with the community in the name of transparency. -The primary day-to-day impact this will have on the community is making available structured support around security-related initiatives such as bilateral disclosure agreements, bug bounty programs and audit work. The community and other subteams will have regular opportunity to interact with, provide input to and receive input from the security subteam. +The primary day-to-day impact this will have on the community is making available structured support around security-related initiatives such as bilateral disclosure agreements, bug bounty programs and audit work. The community and other teams will have regular opportunity to interact with, provide input to and receive input from the security team. ## Example -Here we will walk through an example case that highlights how Grin's security subteam would operate, contrasting with the current process. +Here we will walk through an example case that highlights how Grin's security team would operate, contrasting with the current process. -_In the example case, a vulnerability is disclosed to the disclosure contacts listed in Grin's security policy. It is shown that the community gains greater transparency into the decision making process and related outcomes with the proposed security subteam. Additionally the disclosure map is more tightly contained in the proposed process, resulting in less risk of vulnerability exploitation to the community._ +_In the example case, a vulnerability is disclosed to the disclosure contacts listed in Grin's security policy. It is shown that the community gains greater transparency into the decision making process and related outcomes with the proposed security team. Additionally the disclosure map is more tightly contained in the proposed process, resulting in less risk of vulnerability exploitation to the community._ ### Current Process @@ -46,62 +46,62 @@ _In the example case, a vulnerability is disclosed to the disclosure contacts li * Once the vulnerability is mitigated the core team may optionally share some of the details with the community ### Proposed Process -* Grin's security contacts, all members of the security subteam, make initial assessment of the validity of the vulnerability disclosure -* If disclosure determined to be valid, a conversation and plan of action is determined and executed by the security subteam, following Grin's responsible disclosure policy -* Once the vulnerability is mitigated the security subteam must disclose all information that is not still security-critical with the community +* Grin's security contacts, all members of the security team, make initial assessment of the validity of the vulnerability disclosure +* If disclosure determined to be valid, a conversation and plan of action is determined and executed by the security team, following Grin's responsible disclosure policy +* Once the vulnerability is mitigated the security team must disclose all information that is not still security-critical with the community # Reference-level explanation [reference-level-explanation]: #reference-level-explanation -Grin's security subteam will be implemented as stated below. +Grin's security team will be implemented as stated below. ## Membership * Bootstrapped by existing security contacts (@lehnberg, @hashmap, @joltz) * Minimum three, maximum five members -* Membership is recommended by the security subteam and approved by the core team -* There must be at least one well-represented core developer on the subteam, chosen by the core team +* Membership is recommended by the security team and approved by the core team +* There must be at least one well-represented core developer on the team, chosen by the core team * All members must have relevant domain expertise in security, cryptography, the Grin codebase or Grin community * All members must have high competency in maintaining personal operational security (keeping private keys secure, maintaining secure communications channels etc.) -* Membership is revoked by failure to update the canary, unanimous decision by subteam or decision from the core team - * Note if the core team engages in removing members from the security subteam without reasonable justification, the security subteam loses purpose, becomes ineffective and is relegated to useless security theater +* Membership is revoked by failure to update the canary, unanimous decision by team or decision from the core team + * Note if the core team engages in removing members from the security team without reasonable justification, the security team loses purpose, becomes ineffective and is relegated to useless security theater ### Core Representation -The core team is responsible for choosing one developer who they trust to represent the core interests in the security subteam. This ensures that the core team has adequate representation in security related decisions, without overpowering the majority of discussion. Additionally, with the veto rules defined below, we ensure that in cases of disagreement the security subteam cannot make an impactful decision without the core team's awareness and input. +The core team is responsible for choosing one developer who they trust to represent the core interests in the security team. This ensures that the core team has adequate representation in security related decisions, without overpowering the majority of discussion. Additionally, with the veto rules defined below, we ensure that in cases of disagreement the security team cannot make an impactful decision without the core team's awareness and input. ## Decision Making -* Decisions are made primarily with consideration to the security priorities outlined prior to the subteam's formation -* Unanimous approval of security subteam members is needed to make all impactful decisions +* Decisions are made primarily with consideration to the security priorities outlined prior to the team's formation +* Unanimous approval of security team members is needed to make all impactful decisions * Some decisions made must also be approved by the core team to be carried out: * Adding/removing parties from disclosure agreements - * Adding members to the security subteam + * Adding members to the security team * Updating the security policy - * If the core team does not follow a security subteam decision that it has final approval on, a reasonable explanation must be provided to the security subteam + * If the core team does not follow a security team decision that it has final approval on, a reasonable explanation must be provided to the security team * Some decisions must be made without fully informing the core team of all details: * When a vulnerability requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard) * In these cases not all information will be shared with core team, such as specific details to carry out an exploit, full impact and other relevant information that will create unnecessary risk if disclosed - * The core team must be notified of all cases of these decisions within a reasonable time frame that does not pose a security risk (e.g. after a fix is implemented) and provided with a reasonable explanation by the security subteam + * The core team must be notified of all cases of these decisions within a reasonable time frame that does not pose a security risk (e.g. after a fix is implemented) and provided with a reasonable explanation by the security team ### Vetoes -* Any member of the security subteam may veto an impactful decision +* Any member of the security team may veto an impactful security team decision -The ability to veto an impactful decision is important to preserve the ability for members of the security subteam to make honest recommendations. Important and controversial decisions will be made and members of the security subteam need the ability to not sign their names to a decision they are not comfortable with that could have serious ramifications. These cases will be handled according to the Deadlocks section below. +The ability to veto an impactful security team decision is important to preserve the ability for members of the security team to make honest recommendations. Important and controversial decisions will be made and members of the security team need the ability to not sign their names to a decision they are not comfortable with that could have serious ramifications. These cases will be handled according to the Deadlocks section below. -The security subteam should always strive to reach its own decisions. Constant vetoes and inability to reach unanimous decisions will dampen the advantages of the existence of the team and may render it obsolete. It is important to strike the right balance of different perspectives, expertise and reasonability in members to take advantage of the benefits that come with a diverse and functional security-related decision making process. +The security team should always strive to reach its own decisions. Constant vetoes and inability to reach unanimous decisions will dampen the advantages of the existence of the team and may render it obsolete. It is important to strike the right balance of different perspectives, expertise and reasonability in members to take advantage of the benefits that come with a diverse and functional security-related decision making process. ### Deadlocks -* Impactful decisions that can't be made unanimously by the security subteam will be made by the core team according to their decision making process +* Impactful security team decisions that can't be made unanimously by the security team will be made by the core team according to their decision making process * If the core team is deadlocked, the decision made will be whichever decision, if any, will provide the end user with the greatest measure of security -Deadlocks should be avoided at all costs. The inability of the security subteam to reach a unanimous decision based on the security priorities of Grin's community and core team should be a red flag to the community and core team for the health of the subteam. All members should be able to clearly discuss the factual merits of the options and reach a decision that is in the clear best possible interest for the security of Grin's ecosystem and users. +Deadlocks should be avoided at all costs. The inability of the security team to reach a unanimous decision based on the security priorities of Grin's community and core team should be a red flag to the community and core team for the health of the team. All members should be able to clearly discuss the factual merits of the options and reach a decision that is in the clear best possible interest for the security of Grin's ecosystem and users. ## Example -_By following the previous example in more detail, critical differences can be further explored between the current process and the proposed security subteam process in the handling of a unique case._ +_By following the previous example in more detail, critical differences can be further explored between the current process and the proposed security team process in the handling of a unique case._ _In cases where the vulnerability is so critical that it requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard), the proposed process reduces the likelihood of the vulnerability being exploited with a focused decision making process and by restricting the details to essential persons only._ @@ -116,10 +116,10 @@ _In cases where the vulnerability is so critical that it requires a [deviation f ### Proposed Process -* Grin's security contacts, all members of the security subteam, make initial assessment of the validity of the vulnerability disclosure +* Grin's security contacts, all members of the security team, make initial assessment of the validity of the vulnerability disclosure * It is determined that the disclosed vulnerability is a critical inflation bug -* _A conversation and plan of action is quickly and unanimously determined and executed on by the security subteam, following Grin's responsible disclosure policy_ -* _Once the vulnerability is mitigated, the security subteam must disclose all information that is not still security-critical with the community_ +* _A conversation and plan of action is quickly and unanimously determined and executed on by the security team, following Grin's responsible disclosure policy_ +* _Once the vulnerability is mitigated, the security team must disclose all information that is not still security-critical with the community_ # Drawbacks [drawbacks]: #drawbacks @@ -129,13 +129,13 @@ _In cases where the vulnerability is so critical that it requires a [deviation f * Adds resource overhead to the project that _must_ be upheld (it is not acceptable to fail to follow through in security processes once established!) * Possibly distracts from other important work- we don't want to spread attention too wide too early * Adds bulk to Grin's governance structure -* The community may trust core more than a subteam to make critical security decisions +* The community may trust core more than a team to make critical security decisions # Rationale and alternatives [rationale-and-alternatives]: #rationale-and-alternatives * One alternative, currently in use, is adding the input of a security expert to the existing core decision making process - * This is better than nothing but we lose a few benefits without a dedicated security subteam: + * This is better than nothing but we lose a few benefits without a dedicated security team: * Strengthened security process: reduced likelihood of exploit leaks * Improved transparency: there is currently little transparency around core's security decision making process * Improved reaction times: informed, thoughtful, decisive actions with input from relevant experts only @@ -146,14 +146,14 @@ _In cases where the vulnerability is so critical that it requires a [deviation f # Prior art [prior-art]: #prior-art -grin-governance[0] and security-process[1] RFCs lay groundwork for the security-subteam RFC. +grin-governance[0] and security-process[1] RFCs lay groundwork for the security-team RFC. # Unresolved questions [unresolved-questions]: #unresolved-questions - Is this the most minimal model that can have long-term success for Grin's security-related decision making process? - Is the single-veto model sufficient for a consistent, timely decision making process? -- If core always has at least one appointed member, and one veto gives the decision to the core team, does this significantly weaken the security subteam's purpose? +- If core always has at least one appointed member, and one veto gives the decision to the core team, does this significantly weaken the security team's purpose? # Future possibilities [future-possibilities]: #future-possibilities @@ -165,7 +165,7 @@ grin-governance[0] and security-process[1] RFCs lay groundwork for the security- * Full time contributors * Development of security tooling -* Security Subteam working groups +* Security team working groups * Red team * Blue team * Audit team From 602818f4639f688c48253128742559db208fac5b Mon Sep 17 00:00:00 2001 From: joltz Date: Tue, 28 Jan 2020 11:45:37 -0800 Subject: [PATCH 53/61] Rename RFC to reflect updated team naming --- text/{0000-security-subteam.md => 0000-security-team.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename text/{0000-security-subteam.md => 0000-security-team.md} (100%) diff --git a/text/0000-security-subteam.md b/text/0000-security-team.md similarity index 100% rename from text/0000-security-subteam.md rename to text/0000-security-team.md From b7223b3e3c9238e5a92213e828433b701f38df87 Mon Sep 17 00:00:00 2001 From: Daniel Lehnberg Date: Mon, 3 Feb 2020 10:38:50 -0500 Subject: [PATCH 54/61] fix: replace 'sub-team' with 'team' as per recent governance decision --- text/0001-rfc-process.md | 19 +++++++++--------- text/0002-grin-governance.md | 36 +++++++++++++++++------------------ text/0003-security-process.md | 6 +++--- 3 files changed, 30 insertions(+), 31 deletions(-) diff --git a/text/0001-rfc-process.md b/text/0001-rfc-process.md index 5a9615a..23bc1e4 100644 --- a/text/0001-rfc-process.md +++ b/text/0001-rfc-process.md @@ -35,7 +35,7 @@ Some changes do not require an RFC: If you submit a pull request to implement a new feature without going through the RFC process, it may be closed with a polite request to submit an RFC first. -### Sub-team specific guidelines +### Team specific guidelines _To be added here once available._ @@ -49,8 +49,7 @@ beforehand, to ascertain that the RFC may be desirable; having a consistent impa Ways to prepare and pave the way for writing and submitting an RFC include discussing the topic or posting "pre-RFCs" to our [forum](https://grin-forum.org) for feedback. -As a rule of thumb, receiving encouraging feedback from long-standing project -contributors, and particularly members of the relevant sub-team (if applicable) is a good indication that the RFC is worth pursuing. +As a rule of thumb, receiving encouraging feedback from long-standing project contributors, and particularly members of the relevant team (if applicable) is a good indication that the RFC is worth pursuing. ## Process description [process-description]: #process-description @@ -71,15 +70,15 @@ In order to make a "substantial" change to Grin, one must first get an RFC merge #### Draft -* Each pull request will be labeled with the most relevant sub-team, which will lead to it being triaged by that team and is assigned a shepherd from this team. The shepherd ensures the RFC progresses through the process and that a decision is reached, but they themselves do not make this decision. +* Each pull request will be labeled with the most relevant team, which will lead to it being triaged by that team and is assigned a shepherd from this team. The shepherd ensures the RFC progresses through the process and that a decision is reached, but they themselves do not make this decision. * As the author, you build consensus and integrate feedback. RFCs that have broad support are much more likely to make progress than those that don't receive any comments. They are encouraged to reach out to the RFC shepherd in particular to get help identifying stakeholders and obstacles. -* The relevant sub-team discuss the RFC pull request, as much as possible in the comment thread of the pull request itself. Offline discussion will be summarized on the pull request comment thread. +* The relevant team discuss the RFC pull request, as much as possible in the comment thread of the pull request itself. Offline discussion will be summarized on the pull request comment thread. * RFCs rarely go through this process unchanged, especially as alternatives and drawbacks are shown. As an author you can make edits, big and small, to the RFC to clarify or change the design,but make changes as new commits to the pull request, and leave a comment on the pull request explaining your changes. Specifically, do not squash or rebase commits after they are visible on the pull request. #### Final Comment Period (FCP) -* At some point, a member of the sub-team will propose a "motion for final comment period" (FCP), along with a *disposition* for the RFC (merge, close, or postpone). -* This step is taken when enough of the tradeoffs have been discussed that the sub-team is in a position to make a decision. That does not require consensus amongst all participants in the RFC thread (which is usually impossible). However, the argument supporting the disposition on the RFC needs to have already been clearly articulated, and there should not be a strong consensus *against* that position outside of the sub-team. Sub-team members use their best judgment in taking this step, and the FCP itself ensures there is ample time and notification for stakeholders to push back if it is made prematurely. +* At some point, a member of the assigned team will propose a "motion for final comment period" (FCP), along with a *disposition* for the RFC (merge, close, or postpone). +* This step is taken when enough of the tradeoffs have been discussed that the team is in a position to make a decision. That does not require consensus amongst all participants in the RFC thread (which is usually impossible). However, the argument supporting the disposition on the RFC needs to have already been clearly articulated, and there should not be a strong consensus *against* that position outside of the team. Team members use their best judgment in taking this step, and the FCP itself ensures there is ample time and notification for stakeholders to push back if it is made prematurely. * For RFCs with lengthy discussion, the motion to FCP is usually preceded by a *summary comment* trying to lay out the current state of the discussion and major tradeoffs/points of disagreement. * The FCP lasts ten calendar days, so that it is open for at least 5 business days. It is also advertised widely (i.e. in [Grin News](https://grinnews.substack.com)). This way all stakeholders have a chance to lodge any final objections before a decision is reached. * In most cases, the FCP period is quiet, and the RFC is either merged or closed. However, sometimes substantial new arguments or ideas are raised, the FCP is canceled, and the RFC goes back into draft mode. @@ -96,13 +95,13 @@ In order to make a "substantial" change to Grin, one must first get an RFC merge * Furthermore, the fact that a given RFC has been accepted and is "active" implies nothing about what priority is assigned to its implementation, nor does it imply anything about whether a developer has been assigned the task of implementing the feature. * While it is not necessary that the author of the RFC also write the implementation, it is by far the most effective way to see an RFC through to completion: authors should not expect that other project contributors will take on responsibility for implementing their accepted feature. * Modifications to "active" RFCs can be done in follow-up pull requests. We strive to write each RFC in a manner that it will reflect the final design of the feature; but the nature of software development means that we cannot expect every merged RFC to actually reflect what the end result will be at the time of implementation. -* In general, once accepted, RFCs should not be substantially changed. Only very minor changes should be submitted as amendments. More substantial changes should be new RFCs, with a note added to the original RFC. Exactly what counts as a "very minor change" is up to the sub-team to decide; check sub-team specific guidelines for more details. +* In general, once accepted, RFCs should not be substantially changed. Only very minor changes should be submitted as amendments. More substantial changes should be new RFCs, with a note added to the original RFC. Exactly what counts as a "very minor change" is up to the team to decide; check team specific guidelines for more details. #### Postponed * Some RFC pull requests are tagged with the "postponed" label when they are closed (as part of the rejection process). * An RFC closed with "postponed" is marked as such because we want neither to think about evaluating the proposal nor about implementing the described feature until some time in the future, and we believe that we can afford to wait until then to do so. -* Postponed pull requests may be re-opened when the time is right. We don't have any formal process for that, you should ask members of the relevant sub-team. +* Postponed pull requests may be re-opened when the time is right. We don't have any formal process for that, you should ask members of the relevant team. * Usually an RFC pull request marked as "postponed" has already passed an informal first round of evaluation, namely the round of "do we think we would ever possibly consider making this change, as outlined in the RFC pull request, or some semi-obvious variation of it." (When the answer to the latter question is "no", then the appropriate response is to close the RFC, not postpone it.) #### Closed @@ -113,7 +112,7 @@ In order to make a "substantial" change to Grin, one must first get an RFC merge In the spirit of the proposed process itself, a future "substantial" overhaul to the RFC process should be opened as a new RFC rather than making edits to this RFC. Minor changes can be made by opening pull requests against this document. -As the RFC process is something that should be consistent across all sub-teams and the project as a whole, changes to the process fall under Core's remit. As they evaluate proposals to modify the process, they are expected to consult with sub-teams, and other stakeholders using or being affected by the process. +As the RFC process is something that should be consistent across all teams and the project as a whole, changes to the process fall under Core's remit. As they evaluate proposals to modify the process, they are expected to consult with teams, and other stakeholders using or being affected by the process. ## Drawbacks [drawbacks]: #drawbacks diff --git a/text/0002-grin-governance.md b/text/0002-grin-governance.md index 831eebe..3fcd0df 100644 --- a/text/0002-grin-governance.md +++ b/text/0002-grin-governance.md @@ -12,7 +12,7 @@ Evolve Grin's governance: - Define general community principles; - Convert council into core team and define its responsibilities and processes; - Introduce RFC process; -- Introduce self-governing sub-teams that steward and guide work in each of their focus areas. +- Introduce self-governing teams that steward and guide work in each of their focus areas. # Motivation [motivation]: #motivation @@ -70,11 +70,11 @@ The core team leads the wider Grin project as a whole. In particular: * **Sets overall direction and vision.** Values and philosophies. Steering towards use cases and targets. * **Sets priorities and plans releases.** Maintains high level planning, roadmaps, focus areas, decides on pace and the release schedule. -* **Work on broader, cross-sectional issues.** What falls in-between sub-teams, taking a global view. -* **Adds and removes sub-teams.** While proposals for new sub-teams can come from anywhere, the core team is responsible to ensure structures are productive and make sense. -* **Appoints initial sub-team leadership.** Once a new sub-team is established, core appoints a leader that has the responsibility to grow the sub-team. +* **Work on broader, cross-sectional issues.** What falls in-between teams, taking a global view. +* **Adds and removes teams.** While proposals for new teams can come from anywhere, the core team is responsible to ensure structures are productive and make sense. +* **Appoints initial team leadership.** Once a new team is established, core appoints a leader that has the responsibility to grow the team. * **Responsible for security.** Handles disclosures, vulnerabilities, audits, processes. -* **Handles multi-sig keys and takes high level spending decisions.** Spending proposals can be made by anyone, and sub-teams can have their own own budgets to deal with as they please. +* **Handles multi-sig keys and takes high level spending decisions.** Spending proposals can be made by anyone, and teams can have their own own budgets to deal with as they please. #### Organization & Processes @@ -87,26 +87,26 @@ The core team leads the wider Grin project as a whole. In particular: - The term of core members is currently undefined but may change in the future. - A core member can resign by notifying the rest of the team. - If unreachable for 30 days without any news, a core member can be removed through a core decision. -- The core team *does not* make rulings on RFCs of other sub-teams, although individual team members might also participate in the discussions of sub teams or be members of those teams. +- The core team *does not* make rulings on RFCs of other teams, although individual team members might also participate in the discussions of sub teams or be members of those teams. - Meeting notes should be published and made transparent to the community on a regular basis. ### RFC process An RFC is a Request For Comments document, outlining a proposed improvement or design change to an area of Grin. The exact specifics for the template is TBD. They are kept in their own dedicated repo and need to be accepted before a pull request is merged. Their purpose is to outline a standardized way of making proposals and allow the community to discuss and evaluate whether something is worth doing. Having an RFC accepted means that there's support "in theory" for the suggestion. It does not mean that a change becomes implemented automatically or in the exact way it is proposed, it is high-level design. The work still needs to be carried out. Accepted RFCs guide the broader planning work. -### Sub-teams +### Teams #### Overview -Sub teams are groups organized around specific areas or knowledge fields. They are responsible for these areas, but do not do all the work. Anyone can contribute anywhere, and do not need to hold a particular title to do so. +Teams are groups organized around specific areas or knowledge fields. They are responsible for these areas, but do not do all the work. Anyone can contribute anywhere, and do not need to hold a particular title to do so. -Rather, sub-teams work on policies, processes, and workflows for their specific areas, as required. They are in charge of the RFC process in their specific field: They determine what requires an RFC in their area, they assign RFC shepherds that guide an RFC through its various stages and ensures the right stakeholders become aware of it and solicit their feedback. Ultimately, sub-teams decide whether an RFC in their area should be accepted or rejected. They are responsible to ensure that each RFC in their area has a tracked status, and that they progress towards an outcome. +Rather, teams work on policies, processes, and workflows for their specific areas, as required. They are in charge of the RFC process in their specific field: They determine what requires an RFC in their area, they assign RFC shepherds that guide an RFC through its various stages and ensures the right stakeholders become aware of it and solicit their feedback. Ultimately, teams decide whether an RFC in their area should be accepted or rejected. They are responsible to ensure that each RFC in their area has a tracked status, and that they progress towards an outcome. -Sub-teams self-organize, but should be inclusive and adhere to community values. They should have a leader, often this leader may be part of the core team. They determine how members get added to the team. They should include area experts, and stakeholders. The decision making process should be consensus-seeking where possible. +Teams self-organize, but should be inclusive and adhere to community values. They should have a leader, often this leader may be part of the core team. They determine how members get added to the team. They should include area experts, and stakeholders. The decision making process should be consensus-seeking where possible. -Sub-teams can be broken down into smaller working groups or sub-teams, permanent or temporary, as required and is seen necessary for them to be productive. +Teams can be broken down into smaller working groups or teams, permanent or temporary, as required and is seen necessary for them to be productive. -Each sub-team has a dedicated section on the forum, they should meet regularly, and keep some notes on what was covered and decided. Decisions do not need to happen in meetings, and could for example be handled asynchronously or in the RFC process. +Each team has a dedicated section on the forum, they should meet regularly, and keep some notes on what was covered and decided. Decisions do not need to happen in meetings, and could for example be handled asynchronously or in the RFC process. Meeting notes should be published and made transparent to the community on a regular basis. @@ -126,22 +126,22 @@ In addition to Core team, the following teams are proposed to be created initial * **Fundraising.** Sponsorships, donations, activities to increase project funds. -* **Moderation.** Code of conduct, handles violations, across all areas of the project. To avoid biases and conflicts of interest, this sub-team _does not_ contain any member of the core team. +* **Moderation.** Code of conduct, handles violations, across all areas of the project. To avoid biases and conflicts of interest, this team _does not_ contain any member of the core team. ### In case of conflicts, disagreement, or dissent -Sub-teams are created and appointed by the core, and core can decide to re-organize team structures and shut down dysfunctional teams. This is a "nuclear" option, such decisions should not be taken lightly. The repercussions of such actions can be worse than the initial situation. +Teams are created and appointed by the core, and core can decide to re-organize team structures and shut down dysfunctional teams. This is a "nuclear" option, such decisions should not be taken lightly. The repercussions of such actions can be worse than the initial situation. -In practice, core team members might be engaging with sub-teams as individual contributors. It is however not expected that "the core team" will become involved in the responsibilities of individual sub-teams. Core sets the overall direction for the project, but should not micro manage sub-teams as this defeats the entire purpose of having these teams in the first place. +In practice, core team members might be engaging with teams as individual contributors. It is however not expected that "the core team" will become involved in the responsibilities of individual teams. Core sets the overall direction for the project, but should not micro manage teams as this defeats the entire purpose of having these teams in the first place. -If there are conflicts within sub-teams, these should ideally be resolved within the sub-teams themselves. If this is not possible and there's contentious disagreements that need outside arbitration, sub-teams can invite core or another sub-team to become involved. +If there are conflicts within teams, these should ideally be resolved within the teams themselves. If this is not possible and there's contentious disagreements that need outside arbitration, teams can invite core or another team to become involved. # Drawbacks [drawbacks]: #drawbacks - Adds a lot more structure. This might create overhead. - Could lead to infighting and conflict between teams. -- Could lead to situations where one sub-team does work that conflicts with other sub-teams, "left hand not talking to the right". +- Could lead to situations where one team does work that conflicts with other teams, "left hand not talking to the right". # Rationale and alternatives @@ -167,7 +167,7 @@ Far too many to all be listed here, but here are some: # Future possibilities [future-possibilities]: #future-possibilities -- Introduce more sub-teams +- Introduce more teams - Introduce an electorate - Introduce terms - Define firmer structures and organizational rules diff --git a/text/0003-security-process.md b/text/0003-security-process.md index 1b030d5..cefda36 100644 --- a/text/0003-security-process.md +++ b/text/0003-security-process.md @@ -173,7 +173,7 @@ subject to the deviations described in the 'Deviations from the Standard' sectio Here we would list any agreements we have with neighboring projects to share vulnerability information in accordance with the standard itself and the 'Deviations from the Standard' section in the adoption of the standard. -Agreements would be made by the core team or security sub-team based on capacity +Agreements would be made by the core team or security team based on capacity to engage and relevance/impact on Grin's ecosystem and to an extent the greater cryptocurrency ecosystem. @@ -327,8 +327,8 @@ relevant to the security process for Grin. # Future possibilities [future-possibilities]: #future-possibilities -- Security sub-team - - If a security sub-team is adopted in the future it would likely maintain these processes +- Security team + - If a security team is adopted in the future it would likely maintain these processes - On-chain governance/bug bounties - Bug bounty program - Formal verification From c7aba9700f3ebf6bc8636cb6bb8a4c1617f9656f Mon Sep 17 00:00:00 2001 From: joltz Date: Mon, 10 Feb 2020 09:47:43 -0800 Subject: [PATCH 55/61] Add unresolved question --- text/0000-security-team.md | 1 + 1 file changed, 1 insertion(+) diff --git a/text/0000-security-team.md b/text/0000-security-team.md index 710eaaa..d3fadaa 100644 --- a/text/0000-security-team.md +++ b/text/0000-security-team.md @@ -151,6 +151,7 @@ grin-governance[0] and security-process[1] RFCs lay groundwork for the security- # Unresolved questions [unresolved-questions]: #unresolved-questions +- What are the security priorities of the Grin community and core team? (e.g. is privacy a higher priority than undetectable inflation?) - Is this the most minimal model that can have long-term success for Grin's security-related decision making process? - Is the single-veto model sufficient for a consistent, timely decision making process? - If core always has at least one appointed member, and one veto gives the decision to the core team, does this significantly weaken the security team's purpose? From c2d11b2257c76534bd4f1915b6720e29d190190e Mon Sep 17 00:00:00 2001 From: mino827 <61026964+mino827@users.noreply.github.com> Date: Fri, 14 Feb 2020 10:27:07 +1300 Subject: [PATCH 56/61] Updated GRIN Forum URL Changed to https://forum.grin.mw --- text/0001-rfc-process.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/text/0001-rfc-process.md b/text/0001-rfc-process.md index 23bc1e4..61c2fb8 100644 --- a/text/0001-rfc-process.md +++ b/text/0001-rfc-process.md @@ -47,7 +47,7 @@ A hastily-proposed RFC can hurt its chances of acceptance. Low quality proposals Although there is no single way to prepare for submitting an RFC, it is generally a good idea to pursue feedback from other project contributors beforehand, to ascertain that the RFC may be desirable; having a consistent impact on the project requires concerted effort toward consensus-building. -Ways to prepare and pave the way for writing and submitting an RFC include discussing the topic or posting "pre-RFCs" to our [forum](https://grin-forum.org) for feedback. +Ways to prepare and pave the way for writing and submitting an RFC include discussing the topic or posting "pre-RFCs" to our [forum](https://forum.grin.mw) for feedback. As a rule of thumb, receiving encouraging feedback from long-standing project contributors, and particularly members of the relevant team (if applicable) is a good indication that the RFC is worth pursuing. From d6295f25e728599dc49fe848ba97a45bfd8d7315 Mon Sep 17 00:00:00 2001 From: joltz Date: Thu, 20 Feb 2020 10:24:58 -0800 Subject: [PATCH 57/61] Improve clarity, update member requirements --- text/0000-security-team.md | 46 +++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/text/0000-security-team.md b/text/0000-security-team.md index d3fadaa..e3297ed 100644 --- a/text/0000-security-team.md +++ b/text/0000-security-team.md @@ -10,7 +10,7 @@ # Summary [summary]: #summary -The security team for Grin will carry out the security priorities of the Grin community and core team, which are to be defined prior to formation of the team. The security team will streamline the security decision making process for the core team while maintaining a careful balance of security and transparency for the community. +The security team for Grin carries out the security priorities of the Grin community and core team, which are to be defined prior to formation of the team. The security team streamlines the security decision making process for the core team while maintaining a careful balance of security and transparency for the community. # Motivation [motivation]: #motivation @@ -19,16 +19,16 @@ The security team for Grin will carry out the security priorities of the Grin co * Dedicated support for security-related initiatives such as bilateral disclosure agreements, bug bounty programs and audit work * Increase confidence in the ecosystem for future contributions -A clear and structured decision making process is needed to fully support the current security process as well as community initiatives such as bilateral disclosure agreements, bug bounty programs and future audit work. Currently this process is maintained by the core team who have many areas of focus, leading to a lack of clarity, timeliness and ability to decisively execute on community security initiatives. +A clear and structured decision making process is needed to fully support the current security process as well as community initiatives such as bilateral disclosure agreements, bug bounty programs and future audit work. This process was previously maintained by the core team who had many areas of focus, leading to a lack of clarity, timeliness and ability to decisively execute on community security initiatives. -The security team will free up core resources to focus on other important areas while expanding dedicated resources to security work, allowing the Grin ecosystem to continue to scale. A clear decision making structure around the security process instills confidence in the greater ecosystem and encourages projects, developers and users to continue to increase contributions of time and resources into Grin. +The security team frees up core resources to focus on other important areas while expanding dedicated resources to security work, allowing the Grin ecosystem to continue to scale. A clear decision making structure around the security process instills confidence in the greater ecosystem and encourages projects, developers and users to continue to increase contributions of time and resources into Grin. # Community-level explanation [community-level-explanation]: #community-level-explanation -Grin's security team is responsible for making security related decisions for the Grin project. This includes, but is not limited to, decisions about the specification and implementation of Grin's security processes. As much as possible Grin's security team carries out the security priorities of the community and core team in a transparent manner. +Grin's security team is responsible for making predefined security related decisions for the Grin project. This includes, but is not limited to, decisions about the specification and implementation of Grin's security processes. As much as possible Grin's security team carries out the security priorities of the community and core team in a transparent manner. -There may be some unique security-related cases that require the security team to make some decisions in private to protect the security of Grin users. In these cases, once it is safe to do so, the security team is responsible for sharing as many details as possible with the community in the name of transparency. +There may be some unique security-related cases that require the security team to make some decisions in private to protect the security of Grin users. In these cases, once it is safe to do so, the security team is responsible for sharing as many details as possible with the community for transparency and accountability purposes. The primary day-to-day impact this will have on the community is making available structured support around security-related initiatives such as bilateral disclosure agreements, bug bounty programs and audit work. The community and other teams will have regular opportunity to interact with, provide input to and receive input from the security team. @@ -38,14 +38,14 @@ Here we will walk through an example case that highlights how Grin's security te _In the example case, a vulnerability is disclosed to the disclosure contacts listed in Grin's security policy. It is shown that the community gains greater transparency into the decision making process and related outcomes with the proposed security team. Additionally the disclosure map is more tightly contained in the proposed process, resulting in less risk of vulnerability exploitation to the community._ -### Current Process +### Previous Process * Grin's security contacts make initial assessment of the validity of the vulnerability disclosure * If disclosure determined to be valid, all details of the disclosure are shared with the entire core team * With a conversation between the entire core team, a plan of action is determined and executed, following Grin's responsible disclosure policy * Once the vulnerability is mitigated the core team may optionally share some of the details with the community -### Proposed Process +### Security Team Process * Grin's security contacts, all members of the security team, make initial assessment of the validity of the vulnerability disclosure * If disclosure determined to be valid, a conversation and plan of action is determined and executed by the security team, following Grin's responsible disclosure policy * Once the vulnerability is mitigated the security team must disclose all information that is not still security-critical with the community @@ -60,27 +60,24 @@ Grin's security team will be implemented as stated below. * Bootstrapped by existing security contacts (@lehnberg, @hashmap, @joltz) * Minimum three, maximum five members * Membership is recommended by the security team and approved by the core team -* There must be at least one well-represented core developer on the team, chosen by the core team * All members must have relevant domain expertise in security, cryptography, the Grin codebase or Grin community * All members must have high competency in maintaining personal operational security (keeping private keys secure, maintaining secure communications channels etc.) * Membership is revoked by failure to update the canary, unanimous decision by team or decision from the core team * Note if the core team engages in removing members from the security team without reasonable justification, the security team loses purpose, becomes ineffective and is relegated to useless security theater -### Core Representation - -The core team is responsible for choosing one developer who they trust to represent the core interests in the security team. This ensures that the core team has adequate representation in security related decisions, without overpowering the majority of discussion. Additionally, with the veto rules defined below, we ensure that in cases of disagreement the security team cannot make an impactful decision without the core team's awareness and input. - ## Decision Making +* Distinction between _security-team decisions_ and _security-related decisions_: + * The security team is responsible for making predefined security-team decisions, not all possible ambiguous security-related decisions * Decisions are made primarily with consideration to the security priorities outlined prior to the team's formation * Unanimous approval of security team members is needed to make all impactful decisions * Some decisions made must also be approved by the core team to be carried out: * Adding/removing parties from disclosure agreements * Adding members to the security team * Updating the security policy - * If the core team does not follow a security team decision that it has final approval on, a reasonable explanation must be provided to the security team + * In the event that the core team does not follow a security team recommendation, a reasonable explanation must be provided as to why the security recommendation was not followed * Some decisions must be made without fully informing the core team of all details: - * When a vulnerability requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard) + * When fixing a vulnerability requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard) * In these cases not all information will be shared with core team, such as specific details to carry out an exploit, full impact and other relevant information that will create unnecessary risk if disclosed * The core team must be notified of all cases of these decisions within a reasonable time frame that does not pose a security risk (e.g. after a fix is implemented) and provided with a reasonable explanation by the security team @@ -103,9 +100,9 @@ Deadlocks should be avoided at all costs. The inability of the security team to _By following the previous example in more detail, critical differences can be further explored between the current process and the proposed security team process in the handling of a unique case._ -_In cases where the vulnerability is so critical that it requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard), the proposed process reduces the likelihood of the vulnerability being exploited with a focused decision making process and by restricting the details to essential persons only._ +_In cases where fixing a vulnerability is so critical that it requires a [deviation from the standard](https://github.com/mimblewimble/grin/blob/master/SECURITY.md#deviations-from-the-standard), the proposed process reduces the likelihood of the vulnerability being exploited with a focused decision making process and by restricting the details to essential persons only._ -### Current Process +### Previous Process * Grin's security contacts make initial assessment of the validity of the vulnerability disclosure * It is determined that the disclosed vulnerability is a critical inflation bug @@ -114,7 +111,7 @@ _In cases where the vulnerability is so critical that it requires a [deviation f * _This conversation may take days or weeks as there is no particular decision making process for security-related issues_ * _Once the vulnerability is mitigated the core team may optionally share some of the details with the community_ -### Proposed Process +### Security Team Process * Grin's security contacts, all members of the security team, make initial assessment of the validity of the vulnerability disclosure * It is determined that the disclosed vulnerability is a critical inflation bug @@ -124,7 +121,7 @@ _In cases where the vulnerability is so critical that it requires a [deviation f # Drawbacks [drawbacks]: #drawbacks -* The core team is already busy +* The core team is already busy and the formation and management of a security team may distract from other pressing and immediate issues * Even though it decentralizes the core team, it potentially creates another point of centralization * Adds resource overhead to the project that _must_ be upheld (it is not acceptable to fail to follow through in security processes once established!) * Possibly distracts from other important work- we don't want to spread attention too wide too early @@ -142,6 +139,12 @@ _In cases where the vulnerability is so critical that it requires a [deviation f * Possibility for security initiatives: clear decision making process for bilateral disclosure agreements, bug bounty programs, audit work * Stronger commitments to the community, ecosystem and other projects: viable with a dedicated, clear decision making process * Long-term scalability: extend capacity for security work to grow beyond the core team's attention capacity +* Another alternative is to rely on an open community to handle all incidents as they come in + * This is less desirable than both the previous process and security team process: + * Vulnerabilities would essentially be public before they can by mitigated + * There is no decision making team so mitigation actions may be slow and contentious + * There is no accountability to ensure that the responsible disclosure process is followed + * This model is much more transparent than other alternatives but comes with the unacceptable associated costs above # Prior art [prior-art]: #prior-art @@ -154,7 +157,6 @@ grin-governance[0] and security-process[1] RFCs lay groundwork for the security- - What are the security priorities of the Grin community and core team? (e.g. is privacy a higher priority than undetectable inflation?) - Is this the most minimal model that can have long-term success for Grin's security-related decision making process? - Is the single-veto model sufficient for a consistent, timely decision making process? -- If core always has at least one appointed member, and one veto gives the decision to the core team, does this significantly weaken the security team's purpose? # Future possibilities [future-possibilities]: #future-possibilities @@ -173,9 +175,13 @@ grin-governance[0] and security-process[1] RFCs lay groundwork for the security- * Bug bounty team * Community outreach team +* Expanding security-team decisions + * Currently security-team decisions are restricted to predefined decisions around the security processes + * In the future security-team decisions can be expanded to include more security-related decisions + # References [references]: #references -[0] https://github.com/mimblewimble/grin-rfcs/blob/master/text/0002-grin-governance.md#sub-teams +[0] https://github.com/mimblewimble/grin-rfcs/blob/master/text/0002-grin-governance.md#teams [1] https://github.com/mimblewimble/grin-rfcs/blob/master/text/0003-security-process.md From 1170846aebfe3eeb25aa457da7d84711fa18abdb Mon Sep 17 00:00:00 2001 From: Daniel Lehnberg Date: Wed, 26 Feb 2020 14:18:12 -0500 Subject: [PATCH 58/61] RFC#0011 --- text/{0000-security-team.md => 0011-security-team.md} | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) rename text/{0000-security-team.md => 0011-security-team.md} (98%) diff --git a/text/0000-security-team.md b/text/0011-security-team.md similarity index 98% rename from text/0000-security-team.md rename to text/0011-security-team.md index e3297ed..ed3e862 100644 --- a/text/0000-security-team.md +++ b/text/0011-security-team.md @@ -2,8 +2,8 @@ - Title: security-team - Authors: [joltz](mailto:joltz@protonmail.com) - Start date: Jan 22 2020 -- RFC PR: Edit if merged: [mimblewimble/grin-rfcs#0000](https://github.com/mimblewimble/grin-rfcs/pull/0000) -- Tracking issue: [Edit if merged with link to tracking github issue] +- RFC PR: [mimblewimble/grin-rfcs#39](https://github.com/mimblewimble/grin-rfcs/pull/39) +- Tracking issue: [mimblewimble/grin-pm/#258](https://github.com/mimblewimble/grin-pm/issues/258) --- From aa192aeb82db0b08fd4d0e5b42a0b480a90a22c0 Mon Sep 17 00:00:00 2001 From: Daniel Lehnberg Date: Wed, 26 Feb 2020 14:20:19 -0500 Subject: [PATCH 59/61] Update README --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 214b33f..21cfc6a 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,7 @@ To begin writing your own RFC or to find out more about the process and the gene | [0008-wallet-state-management](text/0008-wallet-state-management.md) | Improve wallet state management. | [0009-enable-faster-sync](text/0009-enable-faster-sync.md) | Enable faster txhashset sync by changing output MMR commitment | [0010-online-transacting-via-tor](text/0010-online-transacting-via-tor.md) | Define standard for transacting via TOR +| [0011-security-team](text/0011-security-team.md) | Establish Grin Security team ## License From 41d45a80bdcdae02f77f98e81d053dd7a2a8c9a2 Mon Sep 17 00:00:00 2001 From: antiochp <30642645+antiochp@users.noreply.github.com> Date: Mon, 20 Apr 2020 14:36:34 +0100 Subject: [PATCH 60/61] commitments are 33 bytes, not 32 bytes reference to proposed nrd-kernels RFC --- text/0005-variable-size-kernels.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/text/0005-variable-size-kernels.md b/text/0005-variable-size-kernels.md index b92fa43..2e3c542 100644 --- a/text/0005-variable-size-kernels.md +++ b/text/0005-variable-size-kernels.md @@ -22,7 +22,7 @@ We were originally including both fee and lock_height on _every_ kernel in the b Each transaction kernel variant may have associated data. For example, height locked kernels include an associated lock height and non-coinbase kernels have an associated fee. Each kernel variant serializes to a fixed size in bytes but this size may be different for each kernel variants. This allows kernels to be serialized efficiently and provides flexibility to introduce new kernel variants that have additional associated data in the future. -A plain kernel is 105 bytes compared to 113 bytes for a height locked kernel. Omitting the lock height from plain kernels saves approximately 7% in kernel storage costs. +A plain kernel is 106 bytes compared to 114 bytes for a height locked kernel. Omitting the lock height from plain kernels saves approximately 7% in kernel storage costs. These changes affect serialization/deserialization of transaction kernels. Kernels are included in p2p messages for transactions and full blocks. Older nodes will serialize transaction kernels differently so nodes on the network need to be able to handle old and new serialization formats. The kernel MMR is also included in the txhashset state file during fast sync so nodes need to be able to support old and new formats of the state file. @@ -34,12 +34,12 @@ These changes affect serialization/deserialization of transaction kernels. Kerne In protocol version 1 all transaction kernels are serialized using the same structure, regardless of kernel variant. All kernels include 8 bytes for the fee and 8 bytes for the lock_height, even if unused. ``` -features (1 byte) | fee (8 bytes) | lock_height (8 bytes) | excess (32 bytes) | signature (64 bytes) +features (1 byte) | fee (8 bytes) | lock_height (8 bytes) | excess (33 bytes) | signature (64 bytes) 00 | 00 00 00 00 01 f7 8a 40 | 00 00 00 00 00 00 00 00 | 08 b1 ... 22 d8 | 33 11 ... b9 69 ``` -All kernels contain a feature byte to determine the variant. Kernels always include an excess commitment and signature, 32 bytes and 64 bytes respectively. The proposal is to have variant specific data follow the feature byte. +All kernels contain a feature byte to determine the variant. Kernels always include an excess commitment and signature, 33 bytes and 64 bytes respectively. The proposal is to have variant specific data follow the feature byte. The initial feature byte would determine the number of subsequent bytes to read for variant specific data. @@ -49,14 +49,14 @@ The initial feature byte would determine the number of subsequent bytes to read 02 (height locked): following 8 bytes for the fee and additional 8 bytes for the lock height. ``` -This would always be followed by a fixed 32 bytes for the excess commitment and 64 bytes for the kernel signature. +This would always be followed by a fixed 33 bytes for the excess commitment and 64 bytes for the kernel signature. ### Protocol Version 2 (Current Version) Plain kernel, includes fee. ``` -features (1 byte) | fee (8 bytes) | excess (32 bytes) | signature (64 bytes) +features (1 byte) | fee (8 bytes) | excess (33 bytes) | signature (64 bytes) 00 | 00 00 00 00 01 f7 8a 40 | 08 b1 ... 22 d8 | 33 11 ... b9 69 ``` @@ -64,7 +64,7 @@ features (1 byte) | fee (8 bytes) | excess (32 bytes) | signature (64 bytes) Coinbase kernel, no fee, no lock height. ``` -features (1 byte) | excess (32 bytes) | signature (64 bytes) +features (1 byte) | excess (33 bytes) | signature (64 bytes) 01 | 08 b2 ... 15 36 | 08 14 ... 98 96 ``` @@ -72,7 +72,7 @@ features (1 byte) | excess (32 bytes) | signature (64 bytes) Height locked kernel, include fee and lock height. ``` -features (1 byte) | fee (8 bytes) | lock_height (8 bytes) | excess (32 bytes) | signature (64 bytes) +features (1 byte) | fee (8 bytes) | lock_height (8 bytes) | excess (33 bytes) | signature (64 bytes) 02 | 00 00 00 00 00 6a cf c0 | 00 00 00 00 00 00 04 14 | 09 4d ... bb 9a | 09 c7 ... bd 54 ``` @@ -146,12 +146,12 @@ To minimize compatibility issues between wallets we have maintained this existin # Future possibilities [future-possibilities]: #future-possibilities -Support for variable size kernels allows new kernel variants to be introduced in the future. One concrete example of this is [relative kernels][0]. Kernels with relative lock heights will have an associated reference to a prior kernel in addition to a lock height based on that prior kernel. +Support for variable size kernels allows new kernel variants to be introduced in the future. One proposed kernel variant is [No Recent Duplicate (NRD) kernels][0]. Kernels with relative lock heights will have an associated reference to a prior kernel in addition to a lock height based on that prior kernel. # References [references]: #references -* [RFC 0000-relative-kernels][0] [fix link once RFC merged] +* [RFC 0000-nrd-kernels][0] [fix link once RFC merged] * [PR #2734 Support for variable size MMRs][1] * [PR #2824 Protocol version support][2] * [PR #2859 Kernel feature variants][3] From e6d4bffd74f154343652500627e0deb3ceecbaff Mon Sep 17 00:00:00 2001 From: Yeastplume Date: Mon, 25 May 2020 16:21:54 +0100 Subject: [PATCH 61/61] Slate V4 (Compact Slates) (#49) --- README.md | 2 +- text/0012-compact-slates.md | 493 ++++++++++++++++++++++++++++++++++++ 2 files changed, 494 insertions(+), 1 deletion(-) create mode 100644 text/0012-compact-slates.md diff --git a/README.md b/README.md index 21cfc6a..74625ed 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ To begin writing your own RFC or to find out more about the process and the gene | [0009-enable-faster-sync](text/0009-enable-faster-sync.md) | Enable faster txhashset sync by changing output MMR commitment | [0010-online-transacting-via-tor](text/0010-online-transacting-via-tor.md) | Define standard for transacting via TOR | [0011-security-team](text/0011-security-team.md) | Establish Grin Security team - +| [0012-compact-slates](text/0012-compact-slates.md) | Introduce new compact slate format (Slate V4) ## License diff --git a/text/0012-compact-slates.md b/text/0012-compact-slates.md new file mode 100644 index 0000000..f6609f5 --- /dev/null +++ b/text/0012-compact-slates.md @@ -0,0 +1,493 @@ + +- Title: Slate V4 (Compact Slates) +- Authors: [Michael Cordner](mailto:yeastplume@protonmail.com) +- Start date: April 3, 2020 +- RFC PR: [mimblewimble/grin-rfcs#49](https://github.com/mimblewimble/grin-rfcs/pull/49) +- Tracking issue: [mimblewimble/grin-wallet#317](https://github.com/mimblewimble/grin-wallet/issues/317) + +--- + +# Summary +[summary]: #summary + +This RFC describes the changes between version 3 and version 4 of the Slate transaction exchange format, which had the goal of reducing the contents of the Slate to be as minimal as possible. + +# Motivation +[motivation]: #motivation + +Previously, the definition of Slate versions up to V3 had been put together with no regard for its size or/and redundant/irrelevant content. In order to facilitate future exchange method possibilities, it's desirable to ensure the Slate is as compact as possible, particularly on the 'first leg' of a transaction exchange which only actually requires minimal information from the transaction initiator. + +This RFC aims to define the contents of a streamlined "compact" slate by: + +* Removing all redundant or unnecessary Slate fields +* Shortening the names of many Slate fields +* Reducing the size of the Slate to be as minimal as possible at all phases of a transaction +* Defining an optional highly-minimized binary format for Slate output + +Although this RFC doesn't address any particular transaction exchange methods that might be facilitated by this streamlining, one could envisage possibilities such as: + +* An exchange placing the entire initial slate in a QR code +* Encoding the initial slate as an easily-cut-and-paste chunk + +# Community-level explanation +[community-level-explanation]: #community-level-explanation + +There are two basic transaction workflows in a two-party Grin transaction: + +* In the Basic workflow, a sender adds their inputs and change outputs to a transaction, and sends them, along with the fee, the amount and their signature data to the recipient. The recipient adds a output for the amount to the transaction, adds their signature data and returns to the sender, who completes the transaction and posts. + +* In the Invoice workflow, the invoice creator adds a new output to a transaction, and sends the amount and signature data to the payer. The payer adds their inputs and change outputs to the transaction along with their signature data and fee information, then returns to the invoicer, who completes the transaction and posts. + +Although previous versions of the Slate included every party's complete inputs and outputs at every stage of the transaction, it is not technically necessary for the initiator to provide their inputs and outputs to the other party. It suffices for the initiator to store the input/output listing in their local transaction context, and only provide the amount, fee, excess and signature data to the other party. Recognising this, it is possible to ensure that the 'first-leg' of a transaction stage is extremely compact. + +For instance, version 3 of the Slate on transaction initiation may have looked something like the following: + +``` +{ + "version_info": { + "version": 3, + "orig_version": 3, + "block_header_version": 2 + }, + "num_participants": 2, + "id": "0436430c-2b02-624c-2032-570501212b00", + "tx": { + "offset": "d202964900000000d302964900000000d402964900000000d502964900000000", + "body": { + "inputs": [ + { + "features": "Coinbase", + "commit": "087df32304c5d4ae8b2af0bc31e700019d722910ef87dd4eec3197b80b207e3045" + }, + { + "features": "Coinbase", + "commit": "08e1da9e6dc4d6e808a718b2f110a991dd775d65ce5ae408a4e1f002a4961aa9e7" + } + ], + "outputs": [ + { + "features": "Plain", + "commit": "0812276cc788e6870612296d926cba9f0e7b9810670710b5a6e6f1ba006d395774", + "proof": "dcff6175390c602bfa92c2ffd1a9b2d84dcc9ea941f6f317bdd0f875244ef23e696fd17c71df79760ce5ce1a96aab1d15dd057358dc835e972febeb86d50ccec0dad7cfe0246d742eb753cf7b88c045d15bc7123f8cf7155647ccf663fca92a83c9a65d0ed756ea7ebffd2cac90c380a102ed9caaa355d175ed0bf58d3ac2f5e909d6c447dfc6b605e04925c2b17c33ebd1908c965a5541ea5d2ed45a0958e6402f89d7a56df1992e036d836e74017e73ccad5cb3a82b8e139e309792a31b15f3ffd72ed033253428c156c2b9799458a25c1da65b719780a22de7fe7f437ae2fccd22cf7ea357ab5aa66a5ef7d71fb0dc64aa0b5761f68278062bb39bb296c787e4cabc5e2a2933a416ce1c9a9696160386449c437e9120f7bb26e5b0e74d1f2e7d5bcd7aafb2a92b87d1548f1f911fb06af7bd6cc13cee29f7c9cb79021aed18186272af0e9d189ec107c81a8a3aeb4782b0d950e4881aa51b776bb6844b25bce97035b48a9bdb2aea3608687bcdd479d4fa998b5a839ff88558e4a29dff0ed13b55900abb5d439b70793d902ae9ad34587b18c919f6b875c91d14deeb1c373f5e76570d59a6549758f655f1128a54f162dfe8868e1587028e26ad91e528c5ae7ee9335fa58fb59022b5de29d80f0764a9917390d46db899acc6a5b416e25ecc9dccb7153646addcc81cadb5f0078febc7e05d7735aba494f39ef05697bbcc9b47b2ccc79595d75fc13c80678b5e237edce58d731f34c05b1ddcaa649acf2d865bbbc3ceda10508bcdd29d0496744644bf1c3516f6687dfeef5649c7dff90627d642739a59d91a8d1d0c4dc55d74a949e1074427664b467992c9e0f7d3af9d6ea79513e8946ddc0d356bac49878e64e6a95b0a30214214faf2ce317fa622ff3266b32a816e10a18e6d789a5da1f23e67b4f970a68a7bcd9e18825ee274b0483896a40" + } + ], + "kernels": [ + { + "features": "Plain", + "fee": "7000000", + "lock_height": "0", + "excess": "000000000000000000000000000000000000000000000000000000000000000000", + "excess_sig": "00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000" + } + ] + } + }, + "amount": "60000000000", + "fee": "7000000", + "height": "5", + "lock_height": "0", + "ttl_cutoff_height": null, + "payment_proof": null, + "participant_data": [ + { + "id": "0", + "public_blind_excess": "033ac2158fa0077f087de60c19d8e431753baa5b63b6e1477f05a2a6e7190d4592", + "public_nonce": "031b84c5567b126440995d3ed5aaba0565d71e1834604819ff9c17f5e9d5dd078f", + "part_sig": null, + "message": null, + "message_sig": null + } + ] +} +``` + +In version 4 of the Slate format, the initial Slate now looks like the following: + +``` +{ + "ver": "4:3", + "id": "0436430c-2b02-624c-2032-570501212b00", + "sta": "S1", + "off": "0gKWSQAAAADTApZJAAAAANQClkkAAAAA1QKWSQAAAAA=", + "amt": "1000000000", + "fee": "8000000", + "sigs": [ + { + "xs": "A5sKZUTyGG1FogzBtH8ZvGaVVOFFxusytVLN0rdpX2DE", + "nonce": "A1MbUjB5g4fjZ3fTYATVBA8TADHGG77D8gHpzbQTl/IE" + } + ] +} +``` + +While a "return" slate might look something like the following: + +``` +{ + "ver": "4:3", + "id": "0436430c-2b02-624c-2032-570501212b00", + "sta": "S2", + "off": "9fKWSQAAAADTApZJAAAAANQClkkAAAAA1QKWSQAAAAA=", + "sigs": [ + { + "xs": "Ai+QL7rLMl1GC6UmKoMyy9bzsSJPasLzLWzIzjFA1eMY", + "nonce": "AzXE1/dlRaI66gEiTGrQF4h3v4zORFG7Hi5r7/xb0p1k", + "part": "ZJ3SW/zvay4eu1FEzoy/d4gX0GpMIgHqOqJFZffXxDXD7Bt8tc8dVq/p9n0Wei38mVtKTuMaSMNXBSo49TQ06A==" + } + ], + "coms": [ + { + "c": "CfdTW8wOTGEsSs76A90BTpRzo9gixyfx+jJrhvIt//ua", + "p": "geV4V00oPT+QSZBppHhVTqm0EhHTApAUBG+CZIsPdQ1Qj4gzGlPs6KbgVrqssYpcDzLCjDchIHZSd9WLX0QPJwXh2Tt7yA+kep7vkAMgsjjtjkYpDMk+twTztQkNyTPoaFmMbBf2WTxkuT2ohFZqcNS9mnEkW5gwuC/WEDilO9AD+d7lqbApZV0nbkp1zquE6oL9+017gyliTKl4LhYzM8zaRCacaTFjLaWhJAEvR5c0RH37qiLEu1NsVqN3Ik0FXbVqHPoqB5EduDYeB375MKTAdyr+n0XCpj5oKKq+9qXRd3ftbOeB8m+H3IiGX0UTDDU7ackUYF++1J2d+FzswxxGine/eMyizkq6yqq/hkwqp2Lht+N0zgogY6aA67x+w9C03f+ZMxfvSatxBwf392AcLZ2liM/Px45CbeIyoUpotoI+AbN6Rfvip5zIbZBAy0R4+KBxXVeGjrw1hdcvpdH2AROItob2rc7pU3eHZaQj/dcfv2CAkNP7bF3cKuQe55gBCuhFKsneD8xayoX9MwtNcvWgQVrm9hw+1K7OsHrYMJ9fe59JpPC74s0vaE0zSDWpDkh3sf8ru6ADR8CsRoBo2YPVZrCX39sPtMqd7SFS66s/tO8Jn9rDqljDmvdjtRiD28UZitwAZEKxXCU5PztkcKC5FWQdIcVviNdVo5nCYI+GXaMhQDhuTqo5pftGlwjeB1aIDX3K3rxdB8wHuvtQefBTuzKGxW0iumYF2/Uy7oHOwVBCEI+BzJHLmqttYy0gDIgrNZoxTVTl8U8vugVOsk6+MYDvC9LRFQZ7hgheRiCnE1wVka7BCKYsk30h26RwV6LKWLkM0n6HBVX9gIKtNvnyWaC1C7o2KnsTNxo+G3K44LYhc2+hAG3vOaFGP14a" + } + ] +} +``` + +The 'return' slate from the recipient to the originator is expected to be larger, as it must include the recipient's outputs/inputs and proofs (as the party who posts the transaction needs all of this information). However, compacting the slate at each stage of its journey allows for future possibilities that may have been more difficult in previous iterations of the slate format. For instance, an initiator taking Grin payments might be expected to be better equipped to receive Grin transactions. They could present the payer with an invoice transaction for the amount compacted to the size of a QR code, which the payer could scan, accept, and then post back to the invoicer's listening wallet infrastructure. + +Compacting the slate also acts as a minor privacy-enhancer by hiding the initiator's outputs from the other party. + +# Reference-level explanation +[reference-level-explanation]: #reference-level-explanation + +### Slate Definition - JSON + +Entries prefixed with `//` denote fields that may be omitted, as well as their default assumed values. `#` denotes a comment for the purposes of illustration. + +Field ordering is canonical. + +``` +{ +# These fields are always present + "ver": "4:3", + "id": "0436430c-2b02-624c-2032-570501212b00", + "sta": "S1", + "off": "0gKWSQAAAADTApZJAAAAANQClkkAAAAA1QKWSQAAAAA=", + +# These fields may or may not be present +//"num_parts: 2, +//"fee": "8000000", +//"amt": "1000000000", +//"feat": 0, +//"ttl": null, + +# Sigs is always present with at least one entry + "sigs": [ + { + "xs": "A5sKZUTyGG1FogzBtH8ZvGaVVOFFxusytVLN0rdpX2DE", +// "part": null, + "nonce": "A1MbUjB5g4fjZ3fTYATVBA8TADHGG77D8gHpzbQTl/IE" + } + ] + +# These structs may or may not be present + +//"coms": null, +//"proof": null, +//"feat_args": null +} +``` + +A description of all fields and their meanings is as follows: + +### Top-Level Slate Struct + +##### Fields - Always present +* `ver` - The slate version and supported block header version, separated by a `:` +* `id` - The slate's UUID, standard hex-string encoding for UUIDs +* `sta` - 2 character String representing the current stage of the the transaction. See [Status Codes](#status-codes) +* `off` - The running transaction offset total, base-64 encoded. All parties select a random offset at the beginning of the transaction and subtract their offset from the excess value of their outputs. +They then subtract the value of the inputs from the offset when committing to inputs, updating the total offset before sending to the next stage. + +##### Fields - Optional, depending on State and transaction options +* `num_parts` - The number of participants in the transaction, assumed to be 2 if omitted +* `amt` - The transaction amount as a string parseable as a u64. May be omitted on a return journey. +* `fee` - The transaction fee as a string parseable as a u64. May be omitted on a return journey, except during an invoice transaction. +* `feat` - Kernel Features ID. If omitted, kernel is assumed to be Plain (0). If set to 1 or otherwise, any arguments required for a +particular kernel feature set will be found in the `feat_args` struct. +* `ttl` - Time to Live, or block height beyond which wallets should refuse to further process the transaction. Assumed 0 (no ttl) if omitted +from the slate. To be used when delayed transaction posting is desired. + +##### Structs - Always present +* `sigs` - An array of signature data containing the signature information of the last participant. See [Signature Data](#signature-data) + +##### Structs - Optional, depending on state of transaction +* `proof` - An optional payment proof request. See [Payment Proof Data](#payment-proof-data) +* `coms` - The [Transaction](https://github.com/mimblewimble/grin/blob/34ff103bb02bc093fe73d36641eb193f7ef2404f/core/src/core/transaction.rs#L871); is removed from the slate in favour of including this top-level Slate field that can be used to reconstruction the transaction object as expected by the Grin node. See [Transaction Object Fields](#transaction-object-fields) +* `feat_args` - Optional arguments for Kernel features. + +#### Status Codes + +Valid values of the `sta` field and their meanings are: + + * `S1` - Standard (payer to payee) transaction flow, new transaction that incorporates the sender's inputs and change output(s), if any. Sender's signature nonce, excess (with kernel offset) is included and ready for sending to recipient. + * `S2` - Standard transaction flow, recipient has created their outputs(s) and supplied their excess, nonce and partial signature, ready to return to recipient for completion. + * `S3` - Standard transaction flow, Slate complete, contains all inputs, outputs and final signatures, and is ready for posting + * `I1` - Invoice (payee to payer) transaction flow, new transaction that incorporates the Invoicer's output(s). Invoicer's signature nonce and excess (with kernel offset) is included and ready for sending to Invoicee. + * `I2` - Invoice workflow, Invoicee has added their inputs and change output(s). Invoicee's signature, nonce and excess is included and ready for sending back to the invoicer. + * `I3` - Invoice workflow, Slate complete, contains all inputs, outputs and final signatures, and is ready for posting + +#### Signature Data + +An entry in the `sigs` array is as follows: + +``` +{ + "xs": "A5sKZUTyGG1FogzBtH8ZvGaVVOFFxusytVLN0rdpX2DE", + "part": null, + "nonce": "A1MbUjB5g4fjZ3fTYATVBA8TADHGG77D8gHpzbQTl/IE" +} +``` + +The `sigs` struct contains is comprised of an array of participant signature data, with each entry comprising: + * `xs` - Base64 encoded short form public key on the secp256k1 curve representing the public blind excess for the participants outputs subtracted from each party's part of the offset. + * `part` - Base64 encoded Aggregated (Schnorr) secp2561k signature represeting the participant's partial sig. May be omitted if the participant does not yet have enough data to create it + * `nonce` - Base64 encoded The public key of the nonce chosen by the participant for their partial signature + +The other party's `sig` entry is removed from the slate before sending it back to the transaction initiator during the S2 and I2 phases. + +#### Payment Proof Data + +If included, the proof structure is: + +``` + "proof": { + "saddr": "7e008eb593ba17d116e282d6267a3c6aad87b910933ad34dfa4d7d2c92b6ba31", +// "rsig": null, + "raddr": "3a425bd5da5f0f78593251ede7fad0ecf7a95679d84b2cb405255d97ce068234" + } +``` + +The `proof` struct is an optional payment proof request that must be filled out by the recipient if requested (only valid for basic transaction flow). Its fields are as follows: + + * `saddr` - Base64 encoded short-form public key on the ed25519 curve, representing the sender's wallet address (see the [payment proofs rfc](https://github.com/mimblewimble/grin-rfcs/blob/master/text/0006-payment-proofs.md) for details. + * `raddr` - Base64 encoded short-form public key on the ed25519 curve represnting the recipient's wallet address + * `rsig` - Base64 encoded EdDSA ed25519 signature representing the recipient's payment-proof signature. Can be omitted if this has not yet been filled out + +#### Transaction Object Fields + +The V4 Version of the Slate contains its own separate representation of Grin's internal Transaction object. (Grin's internal transaction struct definition can be found here: [Transaction](https://github.com/mimblewimble/grin/blob/34ff103bb02bc093fe73d36641eb193f7ef2404f/core/src/core/transaction.rs#L871)). Internally, the wallet will transform its representation into the format expected by the node while posting the transaction. + +The `tx` struct in a V4 Slate is removed, and is replaced instead by the following Top-Level fields. Wallets recreate the transaction object as expected by the Grin node from these fields before posting. + +* `coms` is an array containing the outputs and inputs that have been added to the slate. These will appear during the I1 or S2 phases of a transaction. Each entry includes: + * `f`: The output features, assumed to be 'Plain' if omitted. 0 denotes Plain, 1 denotes Coinbase + * `c`: The output/input commitment, Base64 Encoded + * `p`: The output's range proof, Base64 Encoded. If this is included, the entry is assumed to be an output. If not, it is an input. + +When rebuilding the transaction kernel for the Node (done during the S3 or I3 phases,) the kernel is assumed to be 'Plain' unless the top-level `feat` field is non-zero. In this case, the kernel features are filled accordingly with any needed values from the `feat_args` struct. + +In a typical S3 phase, these fields may look something like: + +``` + "coms": [ + { + "c": "CA8KJv0/AmWQRgvKpTBIwxaE+OTn/TnxaADSkhrSd2yt", + "p": "J+/eWsEWsOljOS5norAUhV9Ygzub4Vf6dxLl71w06sA5c0QHllDeFv0n9UbCPuIL2CoBtLAfgKHAjPV7OjQ5lwSZcK+lfBD3kziiY1iUdjH6ZctAE9D/DdNfQ7a4RdurQDdKbGtihnnnOkHbllutCyV1TDLyvvqyExuwQrK2QkJulhfArQVcBE2txtbWSDjfAAtxraU7UpCMg0MoGRP4v6bm/50hntwTHc9zHegcDblADpMhTNMGh1jlRaCP00n8/Lj520YvZNd0dedynD5U97cfXFTP5C2fjVwwZKCnMZqN90CDAxNuCTA5fFaGTOv2MHMvZrqoVFriuaa+jbjpeU9ifGOWAI5B3HXMeMiSUL0a8c451ohT7zcENCr8jTW+uwA2ejcidMZsL19EaDZX9w0cirdKW5mRI/YPRuw6zBkg8SmXvSLArW0vz94pERcaE0uK9pK/qlgvplmC3/Rv92A9AkyQVeWBieyCoFaxoDL5hGMQaLLc6G3slgCzBUZHX1idNwLhfWAb5mHF97GmqcJf/joEVKuhPpK/mAICLWDUzYQS9gH4GJDlgAz1Bv5KdHY9jo/kB6IDvtxOKnTkl+vK2eK3L178vp02lufgv29fblxSFN9Kw2ujh1j8eCh8RC3RDyAI8qVbHiRDN2N4ANk0fIlc6Gq37D/VChxlTCzd+MmiBJRpqDvYFnz0r2kjuic/xATPQ+G5QoyI30ftXWgLzhGqgZ7S33rywiAP1EsgsoRwUojYvx0skfuasZd+F/Dj2o4RzX32JDcROAh6S49mcr+VumVbPx5tEcZwsQNb6d2yrzCQ6j3x5skaydyaMxhdsMtzb7BOWGWl1GrZ8AALr96R2bdP2vL4mxLtym2qZ/Hh2P5MrnivthS8C+x57TFa" + }, + { + "c": "CA8KJv0/AmWQRgvKpTBIwxaE+OTn/TnxaADSkhrSd2yt" + }, + { + "c": "A43jdi0/AmWQRgvKpTBIwxaE+OTn/TnxaADSkhrxer3f" + } + ], +``` + +#### Feature arguments + +Depending on the chosen Kernel Feature set, `feat_args` may be populated with +arguments specific to the kernel. The exact arguments that will be present here +depend on the value of `feat`. Currently, the only supported kernel is HeightLocked +(value 1) which has the arguments: + +``` +"feat_args": { + "lock_hgt": "2343234" // For HeightLocked kernels (1) +} +``` + +### Changes from existing V3 Slate + +#### Top-Level Slate Struct + +* The `version_info` struct is removed, and is replaced with `ver`, which has the format "[version]:[block header version]" +* `sta` is added, with possible values S1|S2|S3|I1|I2|I3|NA +* `num_participants` is renamed to `num_parts` +* `num_parts` may be omitted from the slate. If omitted its value is assumed to be 2. +* `amount` is renamed to `amt` +* `amt` may be removed from the slate on the S2 phase of a transaction. +* `fee` may be removed from the slate on the S2 phase of a transaction. It may also be omitted when initiating an I1 transaction, and added during the I2 phase. +* `lock_height` is removed +* `feat` is added to the slate denoting the Kernel feature set. May be omitted from the slate if kernel is plain +* `ttl_cutoff_height` is renamed to `ttl` +* `ttl` may be omitted from the slate. If omitted its value is assumed to be 0 (no TTL). +* The `participant_data` struct is renamed to `sigs` +* `tx` is removed +* The `coms` (commitments) array is added, from which the final transaction object can be reconstructed +* The `payment_proof` struct is renamed to `proof` +* The `feat_args` struct is added, which may be populated for non-Plain kernels +* `proof` may be omitted from the slate if it is None (null), +* `offset` is added, which keeps track of the running offset total as it's modified by the participants + +#### Participant Data (`sigs`) + +* `public_blind_excess` is renamed to `xs` +* `public_nonce` is renamed to `nonce` +* `part_sig` is renamed to `part` +* `part` may be omitted if it has not yet been filled out +* `xs` becomes Base64 encoded instead of a hex string +* `nonce` becomes Base64 encoded instead of a hex string +* `part` becomes Base64 encoded instead of a hex string +* `message` is removed +* `message_sig` is removed +* `id` is removed. Parties can identify themselves via the keys stored in their transaction context + +#### Payment Proof Data (`proof`) + +* The `sender_address` field is renamed to `saddr` +* The `receiver_address` field is renamed to `raddr` +* The `receiver_signature` field is renamed to `rsig` +* `saddr` is Base64 encoded instead of a hex string +* `raddr` is Base64 encoded instead of a hex string +* `rsig` is Base64 encoded instead of a hex string +* `rsig` may be omitted if it has not yet been filled out + +### Pretty-Printing + +The examples above are pretty-printed for reference, but all Slates should remove all whitespace before sending to other parties. The whitespace and formatting can be re-added anywhere the Slate needs to be presented for debugging. + +### Slate Definition - Binary + +While the JSON slate remains the first-order slate definition, and should be accepted by all wallets and APIs, wallets should also have the ability to transform slates between the JSON format and a reduced binary format suitable for armoring or direct transfer. The definition of the V4 binary slate format follows. + +All integer values are Big-Endian. + +| Slate V4 Field | type | len | notes | +| ------------------------: | ------ | -------- | ----------------------------------------------------- | +| `ver.slate_version` | u16 | 2 | | +| `ver.block_header_version` | u16 | 2 | | +| `id` | Uuid | 16 | binary Uuid representation | +| `sta` | u8 | 1 | See [Status Byte](#status-byte) | +| `offset` | BlindingFactor | 33 | +| Optional field status | u8 | 1 | See [Optional Field Status](#optional-field-status) | +| `num_parts` | u8 | (1) | If present | +| `amt` | u64 | (4) | If present | +| `fee` | u64 | (4) | If present | +| `feat` | u8 | (1) | If present | +| `ttl` | u64 | (4) | If present | +| `sigs` length | u8 | 1 | Number of entries in the `sigs` struct | +| `sigs` entries | struct | varies | See [Sigs Entries](#sigs-entries) | +| Optional struct status | u8 | 1 | See [Optional Struct Status](#optional-struct-status) | +| `coms` length | u8 | (1) | If present | +| `coms` entries | struct | (varies) | If present. See [Coms Entries](#coms-entries) | +| `proof` | struct | (64) | If present. See [Proof](#proof) | +| `feat_args` entries | struct | (varies) | If present. See [Feature Args](#feature-args) | + +#### Status Byte + +Encodes slate status (`sta`) field, mapped as follows: + +| State | value | +| -------------: | ----- | +| `NA` (Unknown) | 0 | +| `S1` | 1 | +| `S2` | 2 | +| `S3` | 3 | +| `I1` | 4 | +| `I2` | 5 | +| `I3` | 6 | + +#### Optional Field Status + +A bit field that denotes the presence or absence of the optional slate fields. Each bit is +mapped to particular slate field as follows: + +| Bit | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +| ----: | - | - | -------- | ------ | ------ | ----- | ----- | ----------- | +| field | | | | `ttl` | `feat` | `fee` | `amt` | `num_parts` | + +If the corresponding field for a bit is 1, the field is present and must be read accordingly. + +#### Sigs Entries + +Sigs Entries contains a length-prefixed array of entries corresponding to the `sigs` struct. + +| Field | type | len | notes | +| ------: | ------ | ------ | --------------------------- | +| length | u8 | 1 | | +| entries | struct | varies | array of entries, see below | + +Each Sigs Entry is structured as follows: + +| Field | type | len | notes | +| ------------: | -------------------- | ---- | ---------------------------------- | +| complete flag | u8 | 1 | If non-zero, entry contains `part` | +| `xs` | secp256k1 Public Key | 32 | | +| `nonce` | secp256k1 Public Key | 32 | | +| `part` | secp256k1 AggSig | (64) | If present | + +#### Optional Struct Status + +A bit field that denotes the presence or absence of the optional slate structures. Each bit is +mapped to particular slate structure as follows: + +| Bit | 7 | 6 | 5 | 4 | 3 | 2 | 1 | 0 | +| -----: | - | - | - | - | - | - | ------- | ----- | +| struct | | | | | | | `proof` | `coms`| + +If the corresponding field for a struct is 1, the struct is present and must be read accordingly. + +#### Coms Entries + +Coms Entries contains a length-prefixed array of entries corresponding to the `coms` struct. + +| Field | type | len | notes | +| ------: | ------ | ------ | --------------------------- | +| length | u16 | 2 | | +| entries | struct | varies | array of entries, see below | + +Each Coms Entry is structured as follows: + +| Field | type | len | notes | +| ----------: | ---------- | ---- | ----------------------------------------------------- | +| output flag | u8 | 1 | If non-zero, entry is output and contains `p` (proof) | +| `f` | u8 | 1 | features (1 = Coinbase, 0 = Plain) | +| `c` | Commitment | 33 | | +| `p` | Rangeproof | 675 | If present | + +#### Proof + +Optional Payment proof, with fields as follows + +| Proof Field | type | len | notes | +| ----------: | ------------------ | ---- | ------------------------------------ | +| `saddr` | ed25519 Public Key | 32 | | +| `raddr` | ed25519 Public Key | 32 | | +| rsig flag | u8 | 1 | If non-zero, `rsig` field is present | +| `rsig` | ed25519 EDCSA Sig | (64) | If present | + +#### Feature Args + +Optional feature args, presence or absence of which should be determined by the +value of the `feat` field. Currently only present if `feat` is 1. + +| Field | type | len | notes | +| ----------: | ----- | -- | ------------------------------------ | +| `lock_hgt` | u64 | 4 | Lock height, present if `feat` is 1 | + +# Unresolved questions +[unresolved-questions]: #unresolved-questions + +- Is block header version needed? +- Nerd Kernels need to be included + +# Future possibilities +[future-possibilities]: #future-possibilities + +This RFC is envisaged as a necessary first step for all slate-exchange possibilities that would benefit from compactness, e.g: + +* [Slatepack](https://github.com/j01tz/grin-rfcs/blob/slatepack/text/0000-slatepack.md) +* QR Code encoding of slates +* Armored slates