How to properly handle ephemeral state in state channels

[zoedberg]

I've added a test here https://github.com/zoedberg/rgb-integration-tests/tree/ln_transfers that reproduces an issue we encountered on our LN node (run cargo test --test transfers ln_htlc_transfer).

In the test we try to simulate the creation of a commitment TX and of an HTLC TX spending a vout of the commitment. In this case RGB fails to find inputs when calling the contract_assignments_for method. I assume this is because we haven't consumed the fascia. But AFAIR it would be incorrect to consume the fascia of a commitment TX that hasn't been broadcast and might never be. Therefore I'm not sure how we should address this, I normally call contract_assignments_for to obtain the Opout and PersistedState needed to call the add_input method of the TransitionBuilder.

Just a note: to mimic the behavior of LN we had to add some custom methods to create and color a PSBT, this code is mostly copied from RGB repos so we expect it to be correct. We will extend these methods to make also some complex validation tests but for now it should be sufficient to demonstrate the problem.

[dr-orlovsky]

Yes, you can't prepare a new state transition since transition builder uses stash, and you do not add ephemeral state on off chain transactions in the channel to the stash.

The proper way is to add them to the stash via consuming fascia, since otherwise in cases of uncooperative channel closing you will lose funds and won't be able to create a penalty transaction. Yes, this means that stash (and also contract state cache and index) will accumulate growing ephemeral historical state - but that was the part of the design from the very begging, and the stash is made to be capable of doing that.

In other words, the same way you keep and accumulate all signatures for the off chain channel transaction - the same way you have to accumulate fascia in the stash.

[zoedberg]

The proper way is to add them to the stash via consuming fascia, since otherwise in cases of uncooperative channel closing you will lose funds and won't be able to create a penalty transaction. Yes, this means that stash (and also contract state cache and index) will accumulate growing ephemeral historical state - but that was the part of the design from the very begging, and the stash is made to be capable of doing that.

Initially we were consuming the bundles (that are now inside the fascia) but then after some updates we encountered some problems and on telegram (https://t.me/c/1793511697/1041) you said:

So the LN node keeps all the state transitions + anchors (bundles) in its own persistence storage (alongside corresponding signatures) for commitment and htlc transactions never giving them to RGB. Once the channel get closed LN node constructs proper new state transition for a closing or penalty transaction - or uses the most recent bundles from commitment+htlc transactions (in case of unilateral closing) and sends them to RGB ONCE per channel existence.

Anyway, we added a test to the ln_transfers branch, ln_transfers_consume, which consumes the fascia. With this we are able to create the HTLC TX but then we fail to consume the fascia of a later commitment TX that goes back to a previous channel state (simulating an HTLC aborted).

[dr-orlovsky]

Both is true:

1. The stash (+index, +constract state) is designed to keep the excessive, ephemeral state or state with transactions which are lately excluded from blockchain due to re-orgs and so (my answer here)
2. We can deterministically generate state transitions matching previous state (in case of uncooperative closure) so adding them to the stash is not needed in theory (my reply in Telegram)

The problem is that (a) there is no API to do the (2), and (b) designing it represent an immense amount of work.

The only downside of (1) is that it will grow the size of the stash over the time (with channel operations), which can be avoided in case of (2). However, I propose to postpone implementation of (2) for v0.12, since it may require a lot of refactoring and API breaks once again.

[St333p]

If I understand this correctly, it means that consuming the fascia for ephemeral transfers is expected, but it still fails when trying to replace it with a new commitment transaction, as our ln_transfers_consume test shows.

[dr-orlovsky]

Yes, I consider inability to consume following fascia is a bug. It is much easier to fix that comparing to creating builders operating on not just stash but also some additional ephemeral data

[St333p]

Ok so you're going to provide a fix for this before releasing final 0.11, do I get it correctly? Thanks

[dr-orlovsky]

Yes, I will work on the fix. I might discover that it is impossible to have an easy fix and it is easier to refactor the builder. But let's see.

[dr-orlovsky]

The test fails with the error IndexInconsistency(DistinctBundleWitness { bundle_id: BundleId(Array<32>(76b5c47778af7458b03d3a3de704edde172d85e1f1a4b4ffd340e12126619700)), present: Bitcoin(Array<32>(1b594eed0455aa65cc0e5effad92a2802efc8ff9d41aa89dfc7ad55c1a5c386e)), expected: Bitcoin(Array<32>(58248091bab4e09fcc7f91d0ae31f537e8fe0b21677a01f2fc91be38a9690404)) })

Basically this is not a bug and is a safeguard, which happens since existing HTLCs doesn't update their state transitions with new deterministically-generated salts, which results in the fact they have the same RGB state transition bundles and their id, now trying to bind them to a different witness.

The proper way of fixing the problem is to re-generate a new bundle for each HTLCs upon the channel update. Is that possible? Since otherwise I need to remove this safety measure which may reduce off chain wallet security...

[zoedberg]

The issue is not directly related to HTLCs, it happens with commitment transactions (with no HTLCs) as well.

Anyway I don't understand what the check should prevent. I don't see any issue in generating a deterministic bundle ID. Could you please explain what issue you see in allowing this?

[dr-orlovsky]

Normally, if you have a state transition, it must be associated only with one witness bitcoin transaction committing to it. This checks prevents replacing one associations with other.

If you replace it, you won't be able to "go back" and restore funds in case of uncooperative closing.

[zoedberg]

Normally, if you have a state transition, it must be associated only with one witness bitcoin transaction committing to it. This checks prevents replacing one associations with other.

This seems an implementation issue more than a protocol/security issue. Maybe we could store bundles together with the witnesses (e.g. the map key would be (BundleId, WitnessId)) so that we can allow bundles to be associated to more than one witness bitcoin transaction?

If you replace it, you won't be able to "go back" and restore funds in case of uncooperative closing.

By saving RGB amounts for each state update (and assuming a static blinding or deterministic one) I think we can recreate the RGB data that would be necessary on uncooperative closing. So IMO this should not be a concern.
Moreover in this way we could avoid consuming every fascia (or store them somewhere else for later consume), making the disk space needed for every channel update smaller. But this would require a way to extract the Opout and PersistedState from the commitment TX in order to construct the HTLC TX without calling contract_assignments_for, which to me seems the best solution to the problems we've shown via the tests.

[dr-orlovsky]

Yes, this is an implementation issue. I have stated above: while it is possibly to deterministically re-create all RGB-related channel information, it would require a significant work on the rgb-std side, which can/should be postponed to v0.12. In other words: even though this is an implementation issue, it requires a lot of re-implementation work.

So instead we can utilize the ability of the stash to consume all the shit and save our time in v0.11:

Both is true:

The stash (+index, +constract state) is designed to keep the excessive, ephemeral state or state with transactions which are lately excluded from blockchain due to re-orgs and so (my answer here)
We can deterministically generate state transitions matching previous state (in case of uncooperative closure) so adding them to the stash is not needed in theory (my reply in Telegram)
The problem is that (a) there is no API to do the (2), and (b) designing it represent an immense amount of work.

The only downside of (1) is that it will grow the size of the stash over the time (with channel operations), which can be avoided in case of (2). However, I propose to postpone implementation of (2) for v0.12, since it may require a lot of refactoring and API breaks once again.

Does anybody read what I write here?

[dr-orlovsky]

Anyway I have discovered that we need to allow multiple witness transactions for the same bundle, so I will work on that in v.0.11 and you will be able to consume all the fascia for a lightning channel updates: #247

Later I will work on new APIs able to "color" PSBTs from fascia without their consumption into a stash. Maybe I will get it into v0.11, but this will be a low priority and maybe it will get into v0.12

[dr-orlovsky]

So, with #247 merged the proper way of handling ephemeral state in the state channels for v0.11 is the following:

1. Write a custom WitnessResolver which will report WitnessOrd::Archived for all transactions related to all past channel states (since we haven't made new transaction, this means for all off chain channel transactions) and call stock.update_witnesses providing that resolver. This is required in order for the stash to see only state from the funding transaction as an unspent state and properly prepare new commitment transaction.

2. First for commitment transaction, and than for each HTLC transaction:

2.1. Before calling psbt.push_rgb_transition in https://github.com/zoedberg/rgb-integration-tests/blob/0b36bbb286fd8e8758ab58e85e9ac51c1d6c0042/tests/utils/helpers.rs#L1121 call transition.set_nonce, using nonce value 0xFE for the commitment transitions and 0xFF for HTLCs. This will ensure that state from HTLCs are ordered after state from the commitment.

2.2. Consume fascias from all PSBTs for each channel transaction on each of the channel state updates. This will allow to keep the past state necessary for non-cooperative closes

In v0.12 I will work on adding API for using information from fascia alongside stash to generate consignment, which will make steps 1 and 2.2 unnecessary and will save storage space.

[zoedberg]

I'm not sure about the need of WitnessResolver, since the current resolver already returns WitnessOrd::Archived for all offchain TXs.

Anyway I confirm the DistinctBundleWitness error doesn't arise anymore (as expected since the error has been completely removed to support bundles with multiple witness TXs).

It seems though there's still a similar issue:

RGB was provided with an updated operation using different witness transaction. This may happen for instance when some ephemeral state (like a commitment or HTLC transactions in the lightning channels) is added to the stash.
This error means the software uses RGB stash in an invalid way and has business logic bug which has to be fixed.
Operation in stash: OutputAssignment { opout: Opout { op: OpId(Array<32>(37b77555023b0597bc1a3db0d1fd7b3bcaeef9c7e000ff8e24be845e7ae74157)), ty: AssignmentType(4000), no: 0 }, seal: Bitcoin(ExplicitSeal { method: OpretFirst, txid: Array<32>(0171d0dbee9d9964cff589b102b88b6fe5824714002bb79c871579074a9d1388), vout: Vout(0) }), state: RevealedValue { value: Bits64(100), blinding: BlindingFactor(Array<32>(9a02000000000000000000000000000000000000000000000000000000000000)), tag: AssetTag(Array<32>(85b7c6e10375b4159c4081bedeeb745a434292fb7915822097593521fba6b3f0)) }, witness: Some(Bitcoin(Array<32>(0171d0dbee9d9964cff589b102b88b6fe5824714002bb79c871579074a9d1388))) }
New operation: OutputAssignment { opout: Opout { op: OpId(Array<32>(37b77555023b0597bc1a3db0d1fd7b3bcaeef9c7e000ff8e24be845e7ae74157)), ty: AssignmentType(4000), no: 0 }, seal: Bitcoin(ExplicitSeal { method: OpretFirst, txid: Array<32>(81ab4e1c3c250ec2138f81b6eadcdbd1c877fbbc19bd47749e6d1d29309d2869), vout: Vout(0) }), state: RevealedValue { value: Bits64(100), blinding: BlindingFactor(Array<32>(9a02000000000000000000000000000000000000000000000000000000000000)), tag: AssetTag(Array<32>(85b7c6e10375b4159c4081bedeeb745a434292fb7915822097593521fba6b3f0)) }, witness: Some(Bitcoin(Array<32>(81ab4e1c3c250ec2138f81b6eadcdbd1c877fbbc19bd47749e6d1d29309d2869))) }

this panic comes from https://github.com/RGB-WG/rgb-std/blob/master/src/contract/assignments.rs#L75.
I think this check should change or be removed since it's preventing us from consuming the fascia (as the DistinctBundleWitness error was doing).

[dr-orlovsky]

I'm not sure about the need of WitnessResolver, since the current resolver already returns WitnessOrd::Archived for all offchain TXs.

Hm, I believe you need it to report the new commitment transaction as Tentative in order to be able to create HTLCs spending from it.

I think this check should change or be removed

Yes, it was created to prevent from having multiple versions of the same state transition pointing to different witnesses and defining different seals, so now it is unnecessary. Removed in #255

[zoedberg]

Hm, I believe you need it to report the new commitment transaction as Tentative in order to be able to create HTLCs spending from it.

It doesn't seem so, in my tests I'm able to construct the HTLC TXs without setting the commitment to Tentative.

With #255 the test succeeds. I've also added some logic to it to see what happens if a PSBT representing an old state gets broadacasted and by calling update_witnesses that part works as well. I haven't updated RLN with these changes yet, but in the meantime this issue can be closed after merging #255.

[dr-orlovsky]

It doesn't seem so, in my tests I'm able to construct the HTLC TXs without setting the commitment to Tentative.

This is probably due to the fact that you build HTLC state transitions not from the state but directly from PSBTs. I had forgotten that when was writing my comment.

Glad that everything works!