Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWAITING_UNILATERAL bug #2684

Closed
Eskyee opened this issue May 28, 2019 · 9 comments
Closed

AWAITING_UNILATERAL bug #2684

Eskyee opened this issue May 28, 2019 · 9 comments

Comments

@Eskyee
Copy link

Eskyee commented May 28, 2019

AWAITING_UNILATERAL channels stuck and seems. to crash my node. at different times,
my node is over a year and 3 months. old, I guess it seems related to old nodes ,

how c-lightning can knock out my node due to a CPU spike is strange to me;
would love some overstanding on this issue

info:
id: 0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c
alias: btcpayjungle-lightning-node
color: 0265e5
num_peers: 2
num_pending_channels: 1
num_active_channels: 2
num_inactive_channels: 0
address:
- type: ipv4
address: 51.144.112.185
port: 9735
binding:
- type: ipv4
address: 0.0.0.0
port: 9735
version: basedon-v0.7.0-3
blockheight: 578161
network: bitcoin
msatoshi_fees_collected: 11293
fees_collected_msat: 11293msat
peers:

  • id: 022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7
    connected: false
    channels:
    • state: AWAITING_UNILATERAL
      scratch_txid: 8aefc94e1eae524625a1cab2c275c1ce3678d56a89de2e27a4bc22e9ecf0098e
      short_channel_id: 522794x2541x0
      direction: 1
      channel_id: ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a57
      funding_txid: 573a28ca041c1e6423e4e846f382d5d8c33b7fa1fbf0953ac27efd34f6731fae
      private: false
      funding_allocation_msat:
      0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0
      022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7: 5000000
      funding_msat:
      0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0msat
      022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7: 5000000msat
      msatoshi_to_us: 0
      to_us_msat: 0msat
      msatoshi_to_us_min: 0
      min_to_us_msat: 0msat
      msatoshi_to_us_max: 0
      max_to_us_msat: 0msat
      msatoshi_total: 5000000
      total_msat: 5000000msat
      dust_limit_satoshis: 546
      dust_limit_msat: 546000msat
      max_htlc_value_in_flight_msat: '18446744073709551615'
      max_total_htlc_in_msat: 18446744073709551615msat
      their_channel_reserve_satoshis: 0
      their_reserve_msat: 0msat
      our_channel_reserve_satoshis: 546
      our_reserve_msat: 546000msat
      spendable_msatoshi: 0
      spendable_msat: 0msat
      htlc_minimum_msat: 0
      minimum_htlc_in_msat: 0msat
      their_to_self_delay: 144
      our_to_self_delay: 144
      max_accepted_htlcs: 483
      status:
      • 'AWAITING_UNILATERAL:Attempting to reconnect'
        in_payments_offered: 0
        in_msatoshi_offered: 0
        in_offered_msat: 0msat
        in_payments_fulfilled: 0
        in_msatoshi_fulfilled: 0
        in_fulfilled_msat: 0msat
        out_payments_offered: 0
        out_msatoshi_offered: 0
        out_offered_msat: 0msat
        out_payments_fulfilled: 0
        out_msatoshi_fulfilled: 0
        out_fulfilled_msat: 0msat
        htlcs: []
  • id: 02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84
    connected: true
    netaddr:
    • '40.69.71.114:54450'
      global_features: ''
      local_features: '81'
      globalfeatures: ''
      localfeatures: '81'
      channels:
    • state: AWAITING_UNILATERAL
      scratch_txid: d2cb2266ac4b3487d42f57b6e70142222a4c0b283a6de161acc98832868c4e36
      short_channel_id: 567080x257x0
      direction: 0
      channel_id: e53623375ea4a467e5476ddd132d9fd80a95e213aa51b493319e8bd8da6c6564
      funding_txid: 64656cdad88b9e3193b451aa13e2950ad89f2d13dd6d47e567a4a45e372336e5
      private: false
      funding_allocation_msat:
      0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0
      02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84: 100000000
      funding_msat:
      0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c: 0msat
      02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84: 100000000msat
      msatoshi_to_us: 0
      to_us_msat: 0msat
      msatoshi_to_us_min: 0
      min_to_us_msat: 0msat
      msatoshi_to_us_max: 0
      max_to_us_msat: 0msat
      msatoshi_total: 100000000
      total_msat: 100000000msat
      dust_limit_satoshis: 546
      dust_limit_msat: 546000msat
      max_htlc_value_in_flight_msat: '18446744073709551615'
      max_total_htlc_in_msat: 18446744073709551615msat
      their_channel_reserve_satoshis: 1000
      their_reserve_msat: 1000000msat
      our_channel_reserve_satoshis: 1000
      our_reserve_msat: 1000000msat
      spendable_msatoshi: 0
      spendable_msat: 0msat
      htlc_minimum_msat: 0
      minimum_htlc_in_msat: 0msat
      their_to_self_delay: 144
      our_to_self_delay: 144
      max_accepted_htlcs: 483
      status:
      • 'AWAITING_UNILATERAL:Attempting to reconnect'
        in_payments_offered: 0
        in_msatoshi_offered: 0
        in_offered_msat: 0msat
        in_payments_fulfilled: 0
        in_msatoshi_fulfilled: 0
        in_fulfilled_msat: 0msat
        out_payments_offered: 0
        out_msatoshi_offered: 0
        out_offered_msat: 0msat
        out_payments_fulfilled: 0
        out_msatoshi_fulfilled: 0
        out_fulfilled_msat: 0msat
        htlcs: []

My Node!!

getinfo output

{
"id": "0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c",
"alias": "btcpayjungle-lightning-node",
"color": "0265e5",
"num_peers": 3,
"num_pending_channels": 2,
"num_active_channels": 2,
"num_inactive_channels": 0,
"address": [
{
"type": "ipv4",
"address": "51.144.112.185",
"port": 9735
}
],
"binding": [
{
"type": "ipv4",
"address": "0.0.0.0",
"port": 9735
}
],
"version": "basedon-v0.7.0-3",
"blockheight": 578161,
"network": "bitcoin",
"msatoshi_fees_collected": 11293,
"fees_collected_msat": "11293msat"
}

Channel "AWAITING_UNILATERAL" since days. Attempted force closing, CPU 100% this happen about two weeks ago to me,

tried closing out all channels to clear the problem, and force close,
but this has come back again, seems I can't get rid of the last channels

posted this issue in btcpay server GitHub incase others discover this problem,

2019-05-27T14:55:43.696Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13924: Peer connection lost
2019-05-27T14:55:43.697Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13924: Owning subdaemon lightning_openingd died (9)
2019-05-27T15:15:43.527Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13925: Peer connection lost
2019-05-27T15:15:43.528Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13925: Owning subdaemon lightning_openingd died (9)
2019-05-27T15:28:12.532Z lightningd(5): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind estimatesmartfee 100 ECONOMICAL (31251 ms)
2019-05-27T15:28:12.633Z lightningd(5): bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind estimatesmartfee 100 ECONOMICAL exited with status 1
2019-05-27T15:29:20.734Z lightningd(5): bitcoin-cli: finished bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind getblockhash 578048 (80002 ms)
2019-05-27T15:29:28.336Z lightningd(5): bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind getblockhash 578048 exited 1 (after 1 other errors) 'error: Could not connect to the server bitcoind:43782 (error code 1 - "EOF reached")??Make sure the bitcoind server is running and that you are connecting to the correct RPC port.?'
2019-05-27T15:29:53.836Z lightningd(5): FATAL SIGNAL 6 (version basedon-v0.7.0-3)
2019-05-27T15:29:57.136Z lightningd(5): backtrace: common/daemon.c:45 (crashdump) 0x55bd0884276c
2019-05-27T15:30:00.287Z lightningd(5): backtrace: (null):0 ((null)) 0x7f525ed0c0e9
bitcoin-cli -datadir=/etc/bitcoin -rpcconnect=bitcoind getblockhash 578048 exited 1 (after 1 other errors) 'error: Could not connect to the server bitcoind:43782 (error code 1 - "EOF reached")
Make sure the bitcoind server is running and that you are connecting to the correct RPC port.
'
lightningd: Fatal signal 6 (version basedon-v0.7.0-3)
0x55bd08842718 crashdump
common/daemon.c:40
0x7f525ed0c0e9 ???
???:0
Log dumped in crash.log.20190527153000
network=bitcoin added in /root/.lightning/config
alias=btcpayjungle-lightning-node added to /root/.lightning/config
Waiting /root/.nbxplorer/btc_fully_synched to be created...
The chain is fully synched
C-Lightning starting, listening on port 9735
2019-05-27T20:32:59.215Z lightningd(5): --------------------------------------------------
2019-05-27T20:32:59.215Z lightningd(5): Server started with public key 0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c, alias btcpayjungle-lightning-node (color #0265e5) and lightningd basedon-v0.7.0-3
2019-05-27T20:34:20.238Z lightningd(5): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13519: Peer connection lost
2019-05-27T20:34:20.239Z lightningd(5): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13519: Owning subdaemon lightning_openingd died (9)
2019-05-27T20:35:43.955Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13518: Peer connection lost
2019-05-27T20:35:43.955Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13518: Owning subdaemon lightning_openingd died (62208)
2019-05-27T20:45:43.419Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13521: Peer connection lost
2019-05-27T20:45:43.420Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13521: Owning subdaemon lightning_openingd died (62208)

btcpayserver/btcpayserver#839

@Eskyee
Copy link
Author

Eskyee commented May 28, 2019

#2171
#1865
#796
#1865

seems related to a few issues here,

@ZmnSCPxj
Copy link
Collaborator

ZmnSCPxj commented May 28, 2019

AWAITING_UNILATERAL is the force-closing state, it means "we are already doing a force close".

High CPU load is probably not related to the AWAITING_UNILATERAL but if it is, top should report lightning_onchaind as having high CPU load; if it is lightningd or lightning_gossipd, it is (probably) not caused by AWAITING_UNILATERAL.

@Eskyee
Copy link
Author

Eskyee commented May 28, 2019

I managed to close out around 20 channels two weeks ago, force-closed a couple, recovered funds from most channels ok,

but then noticed a stuck peer and two active channels they are not force-closing, as this seems stuck
to me as weeks have gone by,

a few more logs, if that helps

C-Lightning starting, listening on port 9735
2019-05-14T23:52:11.056Z lightningd(5): Restarting onchaind for channel 6091
2019-05-14T23:52:11.056Z lightningd(5): 028303182c9885da93b3b25c9621d22cf34475e63c123942e402ab530c0556e675 chan #6091: Peer permanent failure in ONCHAIN: Funding transaction spent
2019-05-14T23:52:11.056Z lightningd(5): (tx e15db667fb706165f7e7529af151ea294a26f7d3324a95d70a45a90d82f1b327)
2019-05-14T23:52:11.064Z lightningd(5): 028303182c9885da93b3b25c9621d22cf34475e63c123942e402ab530c0556e675 chan #6091: State changed from ONCHAIN to FUNDING_SPEND_SEEN
2019-05-14T23:52:11.632Z lightningd(5): --------------------------------------------------
2019-05-14T23:52:11.632Z lightningd(5): Server started with public key 0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c, alias btcpayjungle-lightning-node (color #0265e5) and lightningd basedon-v0.7.0-3
2019-05-14T23:52:13.108Z lightningd(5): 028303182c9885da93b3b25c9621d22cf34475e63c123942e402ab530c0556e675 chan #6091: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-05-14T23:54:44.073Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13932: Peer connection lost
2019-05-14T23:54:44.073Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13932: Owning subdaemon lightning_openingd died (62208)
2019-05-14T23:55:29.520Z lightningd(5): 0363f6132fa34f0cffcf18b94218cacff7fefebbd085ffdd0cb1bd37d12461f685 chan #9724: State changed from CHANNELD_NORMAL to CHANNELD_SHUTTING_DOWN
2019-05-14T23:57:44.230Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13934: Peer connection lost
2019-05-14T23:57:44.230Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13934: Owning subdaemon lightning_openingd died (62208)
2019-05-15T00:03:50.229Z lightningd(5): 038bcbbbcef1b32a8f566f2ada3c2b15cad089a3e23b259bbcab42fd464fe4770d chan #5814: Peer transient failure in CHANNELD_NORMAL: Reconnected
2019-05-15T00:04:09.713Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from CHANNELD_AWAITING_LOCKIN to CHANNELD_SHUTTING_DOWN
2019-05-15T00:18:58.434Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13935: Peer connection lost
2019-05-15T00:18:58.435Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13935: Owning subdaemon lightning_openingd died (9)
2019-05-15T00:22:46.443Z lightningd(5): 03557fd11b58cb93d2ad4fab4dd4cff7462a97e21e8f6b7a712cb8bd27a96b9eef chan #37: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:22:46.443Z lightningd(5): (tx dd62411faa7318127dc3f278513010db82c54d578b6630eaec7a4343f23d7be9)
2019-05-15T00:22:46.449Z lightningd(5): 03557fd11b58cb93d2ad4fab4dd4cff7462a97e21e8f6b7a712cb8bd27a96b9eef chan #37: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:24:53.131Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:24:53.131Z lightningd(5): (tx 4aceda0d6a00c73f2b00c18c7bb13a9bb8047ca58fd03a1ee6a3bdfdf83c6073)
2019-05-15T00:24:53.140Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:26:49.742Z lightningd(5): 0363f6132fa34f0cffcf18b94218cacff7fefebbd085ffdd0cb1bd37d12461f685 chan #9724: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:26:49.742Z lightningd(5): (tx 5afc33b600eb045ed9fa14e6a3256fbd6492036561e1e43242fca9f69c0fbf3f)
2019-05-15T00:26:49.748Z lightningd(5): 0363f6132fa34f0cffcf18b94218cacff7fefebbd085ffdd0cb1bd37d12461f685 chan #9724: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:29:58.753Z lightningd(5): 023668a30d0a27304695df3fb1af55a4fb75153eac34840817cae0e6a57894fd51 chan #9: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:29:58.753Z lightningd(5): (tx 74b1b9ee3a4938f3d08bc2cae5b0ea77e11920444d75a58caa3375dffcb0944e)
2019-05-15T00:29:58.757Z lightningd(5): 023668a30d0a27304695df3fb1af55a4fb75153eac34840817cae0e6a57894fd51 chan #9: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:30:50.694Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: Peer permanent failure in AWAITING_UNILATERAL: Funding transaction spent
2019-05-15T00:30:50.694Z lightningd(5): (tx 4aceda0d6a00c73f2b00c18c7bb13a9bb8047ca58fd03a1ee6a3bdfdf83c6073)
2019-05-15T00:30:50.696Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from AWAITING_UNILATERAL to AWAITING_UNILATERAL
2019-05-15T00:30:50.696Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from AWAITING_UNILATERAL to FUNDING_SPEND_SEEN
2019-05-15T00:30:51.588Z lightningd(5): 020bf0c421ce915ceb0c1702e136ac3be920c7f6afe3ed75f722babbcb726abd6b chan #13931: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2019-05-15T00:45:20.217Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13936: Peer connection lost
2019-05-15T00:45:20.220Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13936: Owning subdaemon lightning_openingd died (9)
2019-05-15T00:49:20.747Z lightningd(5): 022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7 chan #36: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:49:20.747Z lightningd(5): (tx 8aefc94e1eae524625a1cab2c275c1ce3678d56a89de2e27a4bc22e9ecf0098e)
2019-05-15T00:49:20.752Z lightningd(5): 022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7 chan #36: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:50:23.719Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T00:50:24.429Z lightningd(5): 038b95886d1e9a93b66c7bf782fe279ef41b3b25d65cc9822d836309bfa641f144 chan #25: Peer permanent failure in CHANNELD_SHUTTING_DOWN: Forcibly closed by 'close' command timeout
2019-05-15T00:50:24.429Z lightningd(5): (tx d21f87a726e095a4b1727b5e85b3d4831f82b41585f83d3246e78762af6691df)
2019-05-15T00:50:24.435Z lightningd(5): 038b95886d1e9a93b66c7bf782fe279ef41b3b25d65cc9822d836309bfa641f144 chan #25: State changed from CHANNELD_SHUTTING_DOWN to AWAITING_UNILATERAL
2019-05-15T00:54:27.610Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:04:28.481Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:09:59.473Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:12:34.745Z lightningd(5): lightning_openingd-03ea061d3b061e2a5174ae7ccc09bf47ebeb1d3696f89e24e8d971ba1bed7462d0 chan #13937: Peer connection lost
2019-05-15T01:12:34.746Z lightningd(5): 03ea061d3b061e2a5174ae7ccc09bf47ebeb1d3696f89e24e8d971ba1bed7462d0 chan #13937: Owning subdaemon lightning_openingd died (62208)
2019-05-15T01:15:33.355Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13938: Peer connection lost
2019-05-15T01:15:33.355Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13938: Owning subdaemon lightning_openingd died (62208)
2019-05-15T01:19:00.850Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:22:31.596Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:34:02.978Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T01:47:17.178Z lightningd(5): lightning_openingd-0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13939: Peer connection lost
2019-05-15T01:47:17.179Z lightningd(5): 0318f342854597b2a9af86ae818b794264b3c32e07626ad402a5c47b673482d105 chan #13939: Owning subdaemon lightning_openingd died (9)
2019-05-15T02:06:35.285Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:08:06.289Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:10:37.427Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:14:18.965Z lightningd(5): lightning_openingd-02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13940: Peer connection lost
2019-05-15T02:14:18.966Z lightningd(5): 02efb7d9ddb5e7c05a3b5e6fce5d919390512b4e108e9bce850b206ab39bc32268 chan #13940: Owning subdaemon lightning_openingd died (9)
2019-05-15T02:22:39.466Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:25:43.403Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13941: Peer connection lost
2019-05-15T02:25:43.403Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13941: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:35:43.942Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13942: Peer connection lost
2019-05-15T02:35:43.943Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13942: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:44:41.862Z lightningd(5): lightning_openingd-0313edbe53a41171af2bd1baf76d9095aa58806a05793a91b750d7f02bffe1f38d chan #13943: Peer connection lost
2019-05-15T02:44:41.863Z lightningd(5): 0313edbe53a41171af2bd1baf76d9095aa58806a05793a91b750d7f02bffe1f38d chan #13943: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:45:11.487Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?
2019-05-15T02:45:43.703Z lightningd(5): lightning_openingd-03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13945: Peer connection lost
2019-05-15T02:45:43.703Z lightningd(5): 03ec201d396cc68d34360a13eee019189860006db990bfe85791f42fa9c00e8df1 chan #13945: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:45:46.567Z lightningd(5): lightning_openingd-021067b3b56fdad3985168f4bd5ba31f1c0e3dbcd6c908675f4be5afb99edb7e39 chan #13944: Peer connection lost
2019-05-15T02:45:46.567Z lightningd(5): 021067b3b56fdad3985168f4bd5ba31f1c0e3dbcd6c908675f4be5afb99edb7e39 chan #13944: Owning subdaemon lightning_openingd died (62208)
2019-05-15T02:49:42.275Z lightningd(5): Broadcasting tx 02000000000101ae1f73f634fd7ec23a95f0fba17f3bc3d8d582f346e8e423641e1c04ca283a5700000000006218388000040047304402205a13368d3a091d79ae3f8f7d031770b2a034a1059326598fb1f28c8c1f127d1b022038677b17252377c9f5af8de1c5b37931a7659e47337bdaf00a758fce18b152a301483045022100ecfd4277ac5acf8012e47da652b395a47e80bd2943438dad93ced6e9978a3fd402202eab6d8d098a29846b0f8de39e85c920b91ee993b0b1fd4704bcf2a8bde5798a01475221033b7a005c8ee209426fc70ec707a93bbead4cf9141e6656195cda7349c9b082452103a35c8d0d83d87babfccdf0a302a7f63f267fe9087e8e0e933da328e43e5041b152aec9241220: 26 error code: -26?error message:?bad-txns-vout-empty (code 16)?

btcpayserver_bitcoind btcpayserver/lightning:v0.7.0-3
I gets high cpu spikes when my btcpay server node crash! from c-lightning when Im monitoring with docker stats, strange thing is only happens every now and then!!

@Eskyee
Copy link
Author

Eskyee commented May 28, 2019

Screenshot 2019-05-28 at 09 35 26

I can also check with my spark wallet

@darosior
Copy link
Collaborator

darosior commented May 28, 2019

nits: You can use the <details> markup for logs, it makes the conversation follow-up easier ;)

<details>
<summary>Click me to see logs!</summary>
<p>logs</p>
</details>

About your problem, what was the capacity of your channel ? It seems that the unilateral close transaction is created with no output ( == all capacity is consumed by fees)

@Eskyee
Copy link
Author

Eskyee commented May 28, 2019

https://1ml.com/node/0265e5791c617b6e1dfe04137e71c0c959201592bd7fe3f1bb085c35595b72646c
thanks I will check details
also you can check my node history here, 1ml

@Eskyee
Copy link
Author

Eskyee commented May 28, 2019

https://1ml.com/channel/623511053896056832 been trying to close this channel for the last couple of weeks, also this seem to be the node I can't disconnect AWAITING_UNILATERAL ..
Still a noob with c-lightning but this seem to be my problem node here https://1ml.com/node/022d73890548be3625f822f0294d1c30934c6063df8904bafc70224920e14f10e7

@ZmnSCPxj
Copy link
Collaborator

This is the relevant part:

   error code: -26?error message:?bad-txns-vout-empty (code 16)?

This can happen if the HTLC and channel values are so low that mining fees dominate over it. Looking at your channels:

   msatoshi_total: 5000000
   msatoshi_total: 100000000

The first is 5000 satoshi, or 0.05 mBTC, which is rather small.

The other one is 1mBTC though. And the invalid Bitcoin transaction seems to be only a single one, so it is likely to be the smaller channel.

This bug is already supposedly fixed #632 and #1171 ; this was a spec-level bug I believe.

@cdecker @rustyrussell do you have idea why this is apparently happening again?

Even so, it probably is not related to the CPU load problem, because if we look at timestamps:

   2019-05-15T02:06:35.285Z lightningd(5): Broadcasting tx 
   2019-05-15T02:08:06.289Z lightningd(5): Broadcasting tx 
   2019-05-15T02:10:37.427Z lightningd(5): Broadcasting tx 

It's one and a half minutes between attempts to broadcast, so it should not be the cause of the CPU load.

@Eskyee
Copy link
Author

Eskyee commented May 28, 2019

Thanks for the information
Makes more sense to me now,

I will check my CPU spikes timeline to find the cause of this with my BTCPAY Server

Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants