Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix offer payment failure #7839

Merged
merged 4 commits into from
Nov 20, 2024

Conversation

rustyrussell
Copy link
Contributor

Based on #7610 this fixes a great error which @vincenzopalazzo found (and makes it easier to find similar errors in future, with better logging).

Closes: #7610

@rustyrussell rustyrussell added this to the v24.11 milestone Nov 19, 2024
@vincenzopalazzo
Copy link
Contributor

Uh! and I leave the PR open without fixing it? what a rude person I am becoming. Thanks for taking this for me I completely forgot. LN summit organization took from me some cycles of CPU and I forget to put it on my todo list.


Back to the PR, now a couple of them are failing

2024-11-19T05:18:02.8089117Z test_listchannels_broken_message passed 1 out of the required 1 times. Success!
2024-11-19T05:18:02.8089181Z 
2024-11-19T05:18:02.8089291Z ===End Flaky Test Report===
2024-11-19T05:18:02.8089460Z =========================== short test summary info ============================
2024-11-19T05:18:02.8090180Z FAILED tests/test_pay.py::test_offer_with_private_channels_multyhop2 - TimeoutError: Unable to find "[re.compile('Server started with public key')]" in logs.
2024-11-19T05:18:02.8090559Z FAILED tests/test_xpay.py::test_xpay_simple - AssertionError: Regex pattern did not match.
2024-11-19T05:18:02.8092113Z  Regex: "Failed after 1 attempts\\. Unexpected error \\(invalid_onion_payload\\) from intermediate node: disabling the invoice's blinded path \\(0x0x0/[01]\\) for this payment\\. Then routing failed: We could not find a usable set of paths\\.  The destination has disabled 1 of 1 channels, leaving capacity only 0msat of 10605000msat\\."
2024-11-19T05:18:02.8118172Z  Input: 'RPC call failed: method: xpay, payload: (\'lni1qqgda4y2twmh4tfzndrhv2mjrdfgqq3qn7r7kkqtne03rhppr60mv64mx6venyzylrlpg6qpzc3exdjzsmrppmsrt543ryklhgf5uy89gzr46dnwhj9ux5744fmxhqxqjzeecwja3pwsykt2735tqvm69sppslfcwd9xujwxk034l3vh0dk7ytxpexywt4ktqgpalw2kn6em35wt00r72wcjcwh3zen5zcudux4ks0dufjjgpn99nnsqxwujtgfc3tfpu4duny800lfz9mslcje365ck9znw5txs7my8ms5lylusderu0dhgy8n0ktr2u539njxdyxdsyvwhhvku3uq8wlfcugdypq05ln8pka7h6gcqfhfr5q9qxfecd34dqqegm8mk68605v9hjprt2c256vajne5wxrd34lndlmg3zz2u3jzsl6ml28n9ml53g3uf8jrhhpv5et9pr6dpvggrst89n6l330nas3nhct347geffwve9nk2j4y3lnu22mrvktvauxv4qgylsl44szu7tugacgg7n7mx4weknxveq38clc2xsqgkywfnvs5xcefqxqvx5pvzzqlakpw92vq0wvr3ue7dlrpnajp255yuzk6dqy8f2q4ut433a5xrsjs06q8aqdwjkyvjm7apxnssu4qgwhfkd67ghs6n6k48v6uqczgt88p6tky96qsa0dtx6ayx8usr6llmg3l67rrm9v8spw53pg85c9qfnxkfxndghvpqyvcymt4vavmkj4phwkal20c2nwew066kc334k6hykjr7l58wvz28qppf4ha43qtr34qh879k2gpmuwmuy76nx4h4j8qycf4ypdy5gzc3nqfljld0gk2pjcmyw0twpvg6gez7qvl2gwgggkqxn88ykcg40f8wctj5qqmyay20xys9v5gz5qgsy984dg4g0hug82rpu693y8xrg5hv4kqwpgqr9q0p0hk7pm0drsysj77wwc6j8fgtzt8zus8mqghunqzwjj5n5hgthl8fzn48q5xxm0y2z4gyw37uhgunpgsuqqqqqqgqqqqq5qqtqqqqqqqqqqqqqqqqqqqr36qqqqqqpfqyvu7p70agyqtel9ux3yg845f2la0u3uetytgsy75a29fdcnsncr6pgpl3rdmtd2srqxr2pvppqwpvuk0t7x970kzxwlpwxher999enykwe225j8703ftvdjednhsenuzq52mprh4ywxwpjv6ld2jlzrel5xpvaqyt8rt5t50qw3jj2sxn54pt2wegrv9axl48wz4sunydqw4k9zgvlnl7hfzvkntyxkvdew8549q\',), error: {\'code\': 209, \'message\': "Failed after 1 attempts. We got an error from inside the blinded path 103x2x0/0: we assume it means insufficient capacity. Then routing failed: Actually, I\'m not sure why we didn\'t find the obvious route 103x1x0->103x2x0->0x0x0: perhaps this is a bug?"}'
2024-11-19T05:18:02.8118584Z ERROR tests/test_pay.py::test_offer_with_private_channels_multyhop2 - ValueError: 
2024-11-19T05:18:02.8118674Z Node errors:
2024-11-19T05:18:02.8118872Z  - lightningd-1: Node exited with return code 1
2024-11-19T05:18:02.8119050Z  - lightningd-2: Node exited with return code 1
2024-11-19T05:18:02.8119227Z  - lightningd-3: Node exited with return code 1
2024-11-19T05:18:02.8119407Z  - lightningd-4: Node exited with return code 1
2024-11-19T05:18:02.8119579Z  - lightningd-5: Node exited with return code 1
2024-11-19T05:18:02.8119668Z Global errors:
2024-11-19T05:18:02.8120011Z ERROR tests/test_pay.py::test_offer_with_private_channels_multyhop2 - ValueError: 
2024-11-19T05:18:02.8120095Z Node errors:
2024-11-19T05:18:02.8120273Z  - lightningd-1: Node exited with return code 1
2024-11-19T05:18:02.8120451Z  - lightningd-3: Node exited with return code 1
2024-11-19T05:18:02.8120620Z  - lightningd-2: Node exited with return code 1
2024-11-19T05:18:02.8120794Z  - lightningd-4: Node exited with return code 1
2024-11-19T05:18:02.8120964Z  - lightningd-5: Node exited with return code 1
2024-11-19T05:18:02.8121049Z Global errors:
2024-11-19T05:18:02.8121286Z ERROR tests/test_xpay.py::test_xpay_simple - ValueError: 
2024-11-19T05:18:02.8121373Z Node errors:
2024-11-19T05:18:02.8121519Z  - lightningd-1: had BROKEN messages
2024-11-19T05:18:02.8121692Z Global errors:
2024-11-19T05:18:02.8121922Z ERROR tests/test_xpay.py::test_xpay_simple - ValueError: 
2024-11-19T05:18:02.8122080Z Node errors:
2024-11-19T05:18:02.8122230Z  - lightningd-1: had BROKEN messages
2024-11-19T05:18:02.8122317Z Global errors:
2024-11-19T05:18:02.8122587Z = 2 failed, 650 passed, 162 skipped, 2 xfailed, 4 errors in 1821.60s (0:30:21) =
2024-11-19T05:18:02.8133231Z ##[error]Process completed with exit code 1.
2024-11-19T05:18:02.8218101Z Post job cleanup.
2024-11-19T05:18:02.9176608Z [command]/usr/bin/git version
2024-11-19T05:18:02.9220355Z git version 2.47.0
2024-11-19T05:18:02.9261174Z Temporarily overriding HOME='/home/runner/work/_temp/b51f6472-fbeb-4dd1-afc3-549235d81a44' before making global git config changes
2024-11-19T05:18:02.9261711Z Adding repository directory to the temporary git global config as a safe directory
2024-11-19T05:18:02.9266458Z [command]/usr/bin/git config --global --add safe.directory /home/runner/work/lightning/lightning
2024-11-19T05:18:02.9300287Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-11-19T05:18:02.9349842Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"

@rustyrussell
Copy link
Contributor Author

Yes, this revealed a different bug, so I fixed that too!

rustyrussell and others added 4 commits November 20, 2024 17:35
I had to use fprintf, which is terrible.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Don't reply with update_fail_malformed_htlc, even though WIRE_INVALID_ONION_BLINDING
has BADONION set.  Fail it with a normal error message.

This fixes a known FIXME.

Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Protocol: entry to blinded paths return more useful errors (e.g if it's the final node, you get a real error, otherwise you get invalid_onion_blinding).
```
>           raise RpcError(method, payload, resp['error'])
E           pyln.client.lightning.RpcError: RPC call failed: method: pay, payload: {'bolt11': 'lni1qqgvsykv6pslpmzq73597z0ws2qv5q3qqc3xu3s3rg94nj40zfsy866mhu5vxne6tcej5878k2mneuvgjy8ssqgzpg48getnw30k7enxv4e97amfw3597urjd9mxzar9ta3ksctwdejkcu6ld46kcaredphhqvssacp462c3jt0m5y6wzrj5pp6axehtez7r20265antsrqfpvuu8fwcshgzqjushll8xx9x356tn40gk9mxzkyua9ajtrdpyhm3uaj9nvj0fm9qyqnjp20gp6gr2qsmfas7j086jvkmszqgyys3uht6jq7g4p6vsg7fyyqrx76aulp40m9uxejn57eyczy6v6hqmxr8xx273l480kd5zcl0g9hqp3d9qnsrj40gmeshx0w7fu6j9cfthksz2xv78wxr4ae4wrc3lht8lryc2kxxdpxvs3tpdepm0asuvp0l25fqqvjumjneecjg9etepcu426t2ueu6p3escfrxl9ggnkh5k2vm929tnt26dx66nt67kfy5lgx99py2jhqalaqkyypjeu2artvufgydym4tryv0wvkca78ac64mjeqt70d3wsmjcjgnqnjsyqrzymjxzydqkkw24ufxqslttwlj3s608f0rx2slc7etw0833zgs7kppqd350d9wur2l6mkanmpsswv4xrc49kaq6ey9sfn3rg3z8afgng8fdg8aqr7sxhftzxfdlwsnfcgw2sy8t5mxa0ytcdfat2nkdwqvpy9nnsa9mzzaqtyu8wy0yul9hk026znqy6pn6xd2fpxva7jjcpmvqugeewk7emufyqsru03er082j6ya0p694k6qu858hl0g9rt7g2y042ppzyhdv4qdv99qqs3scf49sff7vg27zlmx6n3kgywrh3s82rwgpza2s8jmqqx72ah0kurp94rj7dlxq438nnm34w78kq7hwu53chx0aqh824eqcsgmq9j2fyvsqttg4yksstnk7h7ga5as69fhemltg0m9hqnn2yxr0lxv70293l7ryqpjfamk2k4mzgax8txef7zcxdzjn0wg7te2cx98ft9cyhk0hquzypasww8m40kyzyqvahtzamflylsygnny5gwqqqqqqyqqqqq2qq9sqqqqqqqqqqqqqqqqqqp6tqqqqqqq5szxdnxnuk5zpctsclfcs4yx0kcvz8nckast2ueswxftuelh49su6sqs2vlzs8ha4gqs9vppqvk0zhg6m8z2prfxa2cerrmn9k803lwx4wukgzlnmvt5xukyjycyauzqwvm6pxlfpfffgktvj3wkurcwrqcp0537hnkd8pnm7tsa0zcklua9zv338cjuphz38wml6tlr8xgdzxdsqh0ks2pns2zkn3c52crfcfs'}, error: {'code': 203, 'message': 'failed: WIRE_INVALID_ONION_PAYLOAD (reply from remote)', 'id': 1, 'failcode': 16406, 'failcodename': 'WIRE_INVALID_ONION_PAYLOAD', 'bolt12': 'lni1qqgvsykv6pslpmzq73597z0ws2qv5q3qqc3xu3s3rg94nj40zfsy866mhu5vxne6tcej5878k2mneuvgjy8ssqgzpg48getnw30k7enxv4e97amfw3597urjd9mxzar9ta3ksctwdejkcu6ld46kcaredphhqvssacp462c3jt0m5y6wzrj5pp6axehtez7r20265antsrqfpvuu8fwcshgzqjushll8xx9x356tn40gk9mxzkyua9ajtrdpyhm3uaj9nvj0fm9qyqnjp20gp6gr2qsmfas7j086jvkmszqgyys3uht6jq7g4p6vsg7fyyqrx76aulp40m9uxejn57eyczy6v6hqmxr8xx273l480kd5zcl0g9hqp3d9qnsrj40gmeshx0w7fu6j9cfthksz2xv78wxr4ae4wrc3lht8lryc2kxxdpxvs3tpdepm0asuvp0l25fqqvjumjneecjg9etepcu426t2ueu6p3escfrxl9ggnkh5k2vm929tnt26dx66nt67kfy5lgx99py2jhqalaqkyypjeu2artvufgydym4tryv0wvkca78ac64mjeqt70d3wsmjcjgnqnjsyqrzymjxzydqkkw24ufxqslttwlj3s608f0rx2slc7etw0833zgs7kppqd350d9wur2l6mkanmpsswv4xrc49kaq6ey9sfn3rg3z8afgng8fdg8aqr7sxhftzxfdlwsnfcgw2sy8t5mxa0ytcdfat2nkdwqvpy9nnsa9mzzaqtyu8wy0yul9hk026znqy6pn6xd2fpxva7jjcpmvqugeewk7emufyqsru03er082j6ya0p694k6qu858hl0g9rt7g2y042ppzyhdv4qdv99qqs3scf49sff7vg27zlmx6n3kgywrh3s82rwgpza2s8jmqqx72ah0kurp94rj7dlxq438nnm34w78kq7hwu53chx0aqh824eqcsgmq9j2fyvsqttg4yksstnk7h7ga5as69fhemltg0m9hqnn2yxr0lxv70293l7ryqpjfamk2k4mzgax8txef7zcxdzjn0wg7te2cx98ft9cyhk0hquzypasww8m40kyzyqvahtzamflylsygnny5gwqqqqqqyqqqqq2qq9sqqqqqqqqqqqqqqqqqqp6tqqqqqqq5szxdnxnuk5zpctsclfcs4yx0kcvz8nckast2ueswxftuelh49su6sqs2vlzs8ha4gqs9vppqvk0zhg6m8z2prfxa2cerrmn9k803lwx4wukgzlnmvt5xukyjycyauzqwvm6pxlfpfffgktvj3wkurcwrqcp0537hnkd8pnm7tsa0zcklua9zv338cjuphz38wml6tlr8xgdzxdsqh0ks2pns2zkn3c52crfcfs', 'raw_message': '40160a0068', 'created_at': 1724699621, 'destination': '032cf15d1ad9c4a08d26eab1918f732d8ef8fdc6abb9640bf3db174372c491304e', 'payment_hash': 'e170c7d38854867db0c11e78b760b573307192be67f7a961cd4010533e281efd', 'status': 'failed', 'amount_msat': 3, 'amount_sent_msat': 0, 'erring_index': 2, 'erring_node': '035d2b1192dfba134e10e540875d366ebc8bc353d5aa766b80c090b39c3a5d885d'}

contrib/pyln-client/pyln/client/lightning.py:416: RpcError
```

Signed-off-by: Vincenzo Palazzo <vincenzopalazzodev@gmail.com>
As we can see from the previous test, l3 tells us why it rejected the payment:

```
lightningd-3 2024-11-19T03:56:27.151Z DEBUG   022d223620a359a47ff7f7ac447c85c46c923da53389221a0054c11c1e3ca31d59-chan#1: Failing HTLC because of an invalid payload (TLV 10 pos 104): cltv_expiry 136 > payment_constraint 130
```

It turns out, we were not updating the block height in the plugin!

Without this, when we create a (non-dummy) blinded path we set a
too-low CLTV restriction, and it doesn't work after a few blocks!

Note we were actually triggering this error in the xpay tests!

Reported-by: Vincenzo Palazzo
Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Fixed: Offers: Receiving bolt12 payments where we have no public channels would fail a few blocks after startup.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to use fprintf, which is terrible.

A tell you a secret, I do it too sometimes :)

Copy link
Contributor

@vincenzopalazzo vincenzopalazzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test_reconnect_no_additional_transient_failure passed 1 out of the required 1 times. Success!
test_offline failed (1 runs remaining out of 2).
	<class 'AssertionError'>
	assert not 'lightningd-1 2024-11-20T09:36:09.764Z DEBUG   connectd: Created listener on 127.0.0.1:46599'
 +  where 'lightningd-1 2024-11-20T09:36:09.764Z DEBUG   connectd: Created listener on 127.0.0.1:46599' = <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fe1acb61b70>>('connectd: Created listener on')
 +    where <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fe1acb61b70>> = <pyln.testing.utils.LightningD object at 0x7fe1acb61b70>.is_in_log
 +      where <pyln.testing.utils.LightningD object at 0x7fe1acb61b70> = <fixtures.LightningNode object at 0x7fe1af59bcd0>.daemon
	[<TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/_pytest/runner.py:341>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/flaky/flaky_pytest_plugin.py:146>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_hooks.py:513>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_manager.py:120>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_callers.py:182>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_result.py:100>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_callers.py:103>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/_pytest/runner.py:177>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/_pytest/runner.py:169>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/_pytest/python.py:1792>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_hooks.py:513>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_manager.py:120>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_callers.py:182>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_result.py:100>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/pluggy/_callers.py:103>, <TracebackEntry /home/runner/.cache/pypoetry/virtualenvs/cln-meta-project-AqJ9wMix-py3.10/lib/python3.10/site-packages/_pytest/python.py:194>, <TracebackEntry /home/runner/work/lightning/lightning/tests/test_connection.py:4509>]
test_offline failed; it passed 0 out of the required 1 times.
	<class 'AssertionError'>
	assert not 'lightningd-1 2024-11-20T09:36:20.916Z DEBUG   connectd: Created listener on 127.0.0.1:39885'
 +  where 'lightningd-1 2024-11-20T09:36:20.916Z DEBUG   connectd: Created listener on 127.0.0.1:39885' = <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fe1ace3e110>>('connectd: Created listener on')
 +    where <bound method TailableProc.is_in_log of <pyln.testing.utils.LightningD object at 0x7fe1ace3e110>> = <pyln.testing.utils.LightningD object at 0x7fe1ace3e110>.is_in_log
 +      where <pyln.testing.utils.LightningD object at 0x7fe1ace3e110> = <fixtures.LightningNode object at 0x7fe1ace3cdf0>.daemon

Not sure if this error should be address here?

tests/test_pay.py Outdated Show resolved Hide resolved
Copy link
Contributor

@vincenzopalazzo vincenzopalazzo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 9932a71

@vincenzopalazzo vincenzopalazzo merged commit 2bf1053 into ElementsProject:master Nov 20, 2024
32 of 39 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants