Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add createonion and sendonion RPC commands #3260

Merged
merged 23 commits into from
Dec 1, 2019

Conversation

cdecker
Copy link
Member

@cdecker cdecker commented Nov 14, 2019

This PR introduces two new RPC methods: createonion and
sendonion. createonion is basically an RPC version of the devtools/onion
CLI tool allowing the caller to create a custom onion. A custom onion can be
used to send arbitrary payloads to hops along a route, and thus allows
experimenting with protocol extensions. The counterpart is the htlc_accepted
hook which allows a plugin implementing the handling of a custom payload
without having to modify the c-lightning source itself.

The sendonion RPC method can be used to initiate a payment using a custom
onion, independently from how or where it was constructed. This was achieved
by splitting sendpay into the onion creation part and the HTLC addition
part, and injecting the onion into the latter. We therefore reuse the entire
payments machinery and can use listsendpays to inspect in-flight and
previous payments initiated with sendonion.

Depending on whether sendonion was given the shared secrets it can decrypt
the onion it receives, and thus also decrypt an eventual error message
returned from a node along the route. If the shared secrets have not been
provided the daemon cannot decrypt the error and any error will result in the
ecrypted error being stored and exposed in listsendpays so the caller, which
knows the shared secrets, can decrypt it externally. This allows oblivious
sending
of payments in which the node only learns what would be revealed to
intermediate hops, but not the length or the nodes in the route.

Use cases

  • Cross-Chain Atomic Swaps: this is the missing piece @bitonic-cjp was
    waiting for. It allows an incoming payment to be intercepted by a plugin on
    one blockchain, and then reinjected into a node running on the other
    blockchain, without having to generate an onion from scratch. createonion
    also allows the inclusion of whatever metadata is necessary to signal to
    the swap node what to do (which chain to swap to, what exchange rate,
    etc.).
  • Experimental protocol extensions: a number of protocol extensions and
    experiments can be based on createonion and sendonion. Here are a few
    examples:
    • Rendezvous routing: the rendez vous draft I wrote requires the ephemeral
      key to be swapped out at the rendez-vous node. The rendez-vous point can
      intercept using htlc_accepted mangle the onion packet as required and
      then re-inject the modified onion into the node. Forwarding is then
      simply handled by the plugin.
    • Trampoline routing: a trampoline waits for incoming trampoline requests
      and then takes over by searching routes to the destination / next
      trampoline and driving retries until the payment succeeds or must fail.
    • Chat: inspired by @joostjager's WhatSat I implemented a chat plugin that
      should be mostly compatible. It uses createonion and sendonion to
      stuff the chat message in the onion payload for the destination.
  • Oblivious payments: Since we can now create an onion externally, i.e., in a
    wallet interface, and inject it, we can hide the route and destination from
    the node itself, reducing the information that a compromised node may learn
    from the node alone.

Open Questions

  • Should createonion and sendonion be developer-only (at least initially)?

Depends #3278
Closes #3227
Fixes #1715

Copy link
Contributor

@rustyrussell rustyrussell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, only minor changes requested.

Also, might want to see if your secret parsing helpers can be used more widely?

Comment on lines -521 to -549
#if !EXPERIMENTAL_FEATURES
if (hop->type != SPHINX_V0_PAYLOAD)
return false;
#endif

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changelog should actually announce TLV! (Changelog-Added: We handle modern TLV-style payloads). This is important because we never announce EXPERIMENTAL features.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hoping you'd add a changelog entry to #3167, since you did all the heavy lifting there, but you're right, that we don't note experimental features in the changelog.

I'd propose the following ling:

Changelog-Added: Plugins may now handle modern TLV-style payloads via the `htlc_accepted` hook

That's a less broad claim than full TLV support for sending and receiving without a plugin, which as you noted is still experimental.

lightningd/pay.c Outdated
else
sp = sphinx_path_new_with_key(cmd, assocdata, session_key);

json_for_each_arr(i, hop, hopstok) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better, BTW, to do the parsing in a new param_hops_array. That way you get proper checking if you use the 'check' rpc command. Might be too awkward though?

lightningd/pay.c Outdated
buffer + pubkeytok->start);


if (!typetok || !json_tok_streq(buffer, typetok, "legacy")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prefer an explicit test here? Either "legacy" or "raw" (the default), rather than a random string. Particularly if we add "tlv" later.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is actually no difference between raw and tlv:

lightning/common/sphinx.c

Lines 526 to 530 in 21555b2

/* Backwards compatibility for the legacy hop_data format. */
if (hop->type == SPHINX_V0_PAYLOAD)
dest[pos++] = 0x00;
else
pos += bigsize_put(dest+pos, raw_size);

Raw really is just confusing and we should remove it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if there's only two options; perhaps we should make this a boolean to flag if is 'legacy'?

otherwise i'd second the idea of defining a strict 'enum' parsing and rejecting unexpected/undefined values.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have an enum for that (enum sphinx_payload_type) so I went ahead and just added an explicit check for legacy or tlv, and added an error message if neither is used,

lightningd/pay.c Outdated

if (hop_count == 0)
return command_fail(cmd, JSONRPC2_INVALID_REQUEST,
"Cannot create an onion without hops.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Onion but no hops? The beer would be terrible!"

CHANGELOG.md Outdated
@@ -529,3 +529,4 @@ There predate the BOLT specifications, and are only of vague historic interest:
[0.3]: https://github.com/ElementsProject/lightning/releases/tag/v0.3-2016-05-26
[0.2]: https://github.com/ElementsProject/lightning/releases/tag/v0.2-2016-01-22
[0.1]: https://github.com/ElementsProject/lightning/releases/tag/v0.1-2015-08-08

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop this?


/* FIXME: Prior to 299b280f7, we didn't put route_nodes and
* route_channels in db. If this happens, it's an old payment,
* so we can simply mark it failed in db and return. */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's OK to delete this (released in v0.6, so over 12 months ago: a payment is unlikely to still be pending since then), but you must place a justification for it in the commit msg. Like this one :)

common/json.c Outdated
{
size_t seclen = sizeof(struct secret), hexlen = tok->end - tok->start;
if (hexlen != 2 * seclen)
return false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hex_decode does this check for you, BTW!

i.e. this function is a one-liner.

}
} else {
path_secrets = NULL;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, it's be nice to implement this as param_secrets_array so 'check' rpc works better.

```

The `onion` corresponds to 1366 hex-encoded bytes. Each shared secret consists
of 23 hex-encoded bytes. Both arguments can be passed on to **sendonion**.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

32 hex-encoded bytes?

doc/lightning-createonion.7.md Outdated Show resolved Hide resolved
@bitonic-cjp
Copy link
Contributor

Nice.

Out of curiosity: will this allow sendpay to be re-implemented as a plugin?

@cdecker
Copy link
Member Author

cdecker commented Nov 18, 2019

Nice.

Out of curiosity: will this allow sendpay to be re-implemented as a plugin?

Indeed that's a possibility. An example that does this can be seen here 05c5146

Copy link
Collaborator

@niftynei niftynei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you inadvertently removed the randomization initialization for the onion padding.

lightningd/pay.c Outdated
buffer + pubkeytok->start);


if (!typetok || !json_tok_streq(buffer, typetok, "legacy")) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if there's only two options; perhaps we should make this a boolean to flag if is 'legacy'?

otherwise i'd second the idea of defining a strict 'enum' parsing and rejecting unexpected/undefined values.

packet = create_onionpacket(cmd, sp, &shared_secrets);
if (!packet)
return command_fail(cmd, LIGHTNINGD,
"Could not create onion packet");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe give a hint as to what input values might have caused the failure?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a whole lot we can do actually, it either failed to generate the keys in generate_hop_params which might happen due to invalid keys, or ecdh failing, or we failed to write an onion frame, e.g., payload was too long.

lightningd/pay.c Outdated Show resolved Hide resolved
lightningd/pay.c Outdated
response = json_stream_success(cmd);
json_add_hex(response, "onion", serialized, tal_bytelen(serialized));
json_array_start(response, "shared_secrets");
for (size_t i=0; i<hop_count; i++) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're really all in on those Golang spacing rules ;)
-> i = 0; i < hop_count;

l1 = node_factory.get_node()

hops = [{
"type": "legacy",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reiterating comment about a flag for 'is_legacy', since that seems to be most of what the type field is used for here.

Copy link
Contributor

@rustyrussell rustyrussell Nov 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that the spec reserves one value for further extensions: that is neither legacy nor TLV.
Perhaps we'll drop legacy before then and it won't be an issue though...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made sure that either the style is not provided (defaulting to TLV) or it is one that we know how to handle (legacy or tlv for now).


/* Check if at destination. */
if (origin_index == tal_count(route_nodes) - 1) {
assert(route_nodes == NULL || origin_index < tal_count(route_nodes));
/* If any channel is to blame, it's the last one. */
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the indentation on this is wrong now since there's no if; but the else statement below is untouched 😕

it seems like this commit would fail to compile.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops, that's what I get for trying to disect a change into separate commits. Fixed in a fixup.

routing_failure->channel_dir =
node_id_idx(&ld->id, &payment->route_nodes[0]);
} else {
routing_failure->erring_channel = NULL;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the channel_dir not need to be set either?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving it unset on purpose so valgrind can tell us if we touch it inadvertently. We use the erring_channel as sentinel for the presence of both, and if we touch the direction we know it's an error (it's used in the JSON-RPC only anyway).

lightningd/pay.c Outdated
erring_channel = NULL;
erring_node = NULL;

/* We don't know if there's another route, that'd depend in
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: depend in -> depend on

@@ -2273,6 +2280,7 @@ wallet_payment_by_hash(const tal_t *ctx, struct wallet *wallet,
", msatoshi_sent"
", description"
", bolt11"
", failonionreply"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this column already exist in the database? i don't see a migration statement in this commit 👀

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does indeed 👍 We just didn't display it so far.


```json
{
"onion": "0003f3f80d2142b953319336d2fe4097[..snip...]6af33fcf4fb113bce01f56dd62248a9e5fcbbfba35c",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

....:scissors:.... :D

Copy link
Member Author

@cdecker cdecker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should change "type" to "style" since we've used that elsewhere. Consistency in APIs is really important.

That's a really weird naming convention if you ask me, but I changed it for the sake of consistency. I'll keep pubkey though.

lightningd/pay.c Show resolved Hide resolved
@cdecker
Copy link
Member Author

cdecker commented Nov 25, 2019

I think you inadvertently removed the randomization initialization for the onion padding.

This branch predates that fix. I have rebased it locally, and checked that the fix is still in there. Will push as soon as I have addressed your feedback 👍

@cdecker
Copy link
Member Author

cdecker commented Nov 25, 2019

@niftynei: GH just decided to forget all the replies I had pending addressing your excellent poitns. I'll try to recreate the important ones from memory.

Pushing the fixups and rebased stack now.

@cdecker
Copy link
Member Author

cdecker commented Nov 25, 2019

@niftynei: GH just decided to forget all the replies I had pending addressing your excellent poitns. I'll try to recreate the important ones from memory.

Nevermind, GH decided to have a reverse lobotomy when I submitted the comment :-)

@cdecker
Copy link
Member Author

cdecker commented Nov 25, 2019

Thanks @niftynei and @rustyrussell for the extensive feedback. I tried to address everything, and the resulting fixup stack is a bit big. Here's the range-diff for an easier comparison: eefd97ff9671b99714305e296e951d4c3075539c..124773bfb14c56e0ea6d474a5872bf3b7581cd72

@cdecker cdecker marked this pull request as ready for review November 25, 2019 14:54
@cdecker
Copy link
Member Author

cdecker commented Nov 25, 2019

As always, happy to squash in the fixup!s once the reviewers have acknowledged the changes (ACKs should carry over automatically unless I managed to break @bitcoin-bot again).

Copy link
Collaborator

@darosior darosior left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really cool!


try:
l1.rpc.waitsendpay(payment_hash=payment_hash)
raise ValueError()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or the else: block can be used but it's strictly the same behaviour.

These are useful for the `createonion` JSON-RPC we're going to build next. The
secret is used for the optional `session_key` while the hex-encoded binary is
used for the `assocdata` field to which the onion commits. The latter does not
have a constant size, hence the raw binary conversion.
This is what provides us with the ability to add custom fields in the payload
when using `createonion` so make sure we actually have access to it.

Changelog-Changed: The TLV payloads for the onion packets are no longer considered an experimental feature and generally available.
Changelog-Added: Plugins may now handle modern TLV-style payloads via the `htlc_accepted` hook

Signed-off-by: Christian Decker <@cdecker>
This allows us to create an onion in the JSON-RPC that we can then later inject with the `sendonion` command that we're about to implement.
If we use `sendonion` we don't actually know the destination, so we
make the destination a pointer which is NULL if we don't know.
This means that c-lightning can now internally decrypt an eventual error
message, and not force the caller to implement the decryption. The main
difficulty was that we now have a new state (channels and nodes not specified,
while shared_secrets are specified) which needed to be handled.
If we can't decode the onion, because the onion got corrupted or we used
`sendonion` without specifying the `shared_secrets` used, the best we can do
is tell the caller instead.
Changelog-Added: Added `createonion` and `sendonion` JSON-RPC methods allowing the implementation of custom protocol extensions that are not directly implemented in c-lightning itself.
We're about to create a param helper for sphinx hops and this struct seems
like the correct place to store the result.
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
If we initiated the payment using an externally generated onion we don't know
what the final hop gets, or even who it is, so we don't display the amount in
these cases. I chose to show `null` instead in order not to break dependees
that rely on the value being there.
@cdecker
Copy link
Member Author

cdecker commented Nov 28, 2019

I rebased and squashed the fixups. The fixups can still be seen here in the form of a range-diff

Ping @rustyrussell and @niftynei 😉

@rustyrussell
Copy link
Contributor

Ack f256bd8

@cdecker cdecker merged commit 0b61781 into ElementsProject:master Dec 1, 2019
cdecker added a commit to cdecker/lightning that referenced this pull request Dec 2, 2019
As suggested by @niftynei here: ElementsProject#3260 (comment)

Suggested-by: Lisa Neigut <@niftynei>
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
cdecker added a commit to cdecker/lightning that referenced this pull request Dec 4, 2019
As suggested by @niftynei here: ElementsProject#3260 (comment)

Suggested-by: Lisa Neigut <@niftynei>
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
cdecker added a commit to cdecker/lightning that referenced this pull request Dec 6, 2019
As suggested by @niftynei here: ElementsProject#3260 (comment)

Suggested-by: Lisa Neigut <@niftynei>
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
cdecker added a commit to cdecker/lightning that referenced this pull request Dec 9, 2019
As suggested by @niftynei here: ElementsProject#3260 (comment)

Suggested-by: Lisa Neigut <@niftynei>
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
cdecker added a commit to cdecker/lightning that referenced this pull request Dec 9, 2019
As suggested by @niftynei here: ElementsProject#3260 (comment)

Suggested-by: Lisa Neigut <@niftynei>
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
cdecker added a commit to cdecker/lightning that referenced this pull request Dec 10, 2019
As suggested by @niftynei here: ElementsProject#3260 (comment)

Suggested-by: Lisa Neigut <@niftynei>
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
cdecker added a commit that referenced this pull request Dec 11, 2019
As suggested by @niftynei here: #3260 (comment)

Suggested-by: Lisa Neigut <@niftynei>
Suggested-by: Rusty Russell <@rustyrussell>
Signed-off-by: Christian Decker <@cdecker>
@Fonta1n3
Copy link

Where is this chat plugin you speak of?

@darosior
Copy link
Collaborator

darosior commented Aug 21, 2020 via email

@Fonta1n3
Copy link

@fontaine it can be found in https://github.com/lightningd/plugins/ along will other plugins.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
Le vendredi, août 21, 2020 5:01 PM, Fontaine notifications@github.com a écrit :

Where is this chat plugin you speak of?


You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.

I don’t see it in there...

@darosior
Copy link
Collaborator

darosior commented Aug 21, 2020 via email

@Fonta1n3
Copy link

Fonta1n3 commented Aug 21, 2020 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Revive "sendpay custom onion" feature
6 participants