Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LND crashed (panic) during QueryRoutes / SendToRouteSync #3712

Closed
LNBIG-COM opened this issue Nov 12, 2019 · 15 comments · Fixed by #3818
Closed

LND crashed (panic) during QueryRoutes / SendToRouteSync #3712

LNBIG-COM opened this issue Nov 12, 2019 · 15 comments · Fixed by #3818
Assignees
Labels
bug Unintended code behaviour multi-hop onion routing

Comments

@LNBIG-COM
Copy link

Background

I had intense code executions QueryRoutes / SendToRouteSync with big lists of ignored_pairs & ignored_nodes. One from LNDs (lnd-01) was crashed

Your environment

  • version of lnd - 0.8.0-beta

Steps to reproduce

May be intense rebalancing through QueryRoutes (with ignored_pairs & ignored_nodes) -> SendToRouteSync executions.

stacktrace:

2019-11-12 12:53:08.431 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:08.819 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.160 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.694 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:09.823 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.453 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.505 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.878 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:10.944 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.204 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.723 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:11.912 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.307 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.447 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.921 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:12.926 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.482 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.620 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.701 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:13.789 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2019-11-12 12:53:14.545 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
panic: runtime error: index out of range [20] with length 20

goroutine 8943415 [running]:
github.com/lightningnetwork/lnd/routing/route.(*Route).ToSphinxPath(0xc00648a140, 0x1da7740, 0xc010fbda00, 0x20)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/route/route.go:306 +0x53a
github.com/lightningnetwork/lnd/routing.generateSphinxPacket(0xc00648a140, 0xc0097def3c, 0x20, 0x20, 0xc01e500030, 0x0, 0xc00bede880, 0xf26542, 0xfc5f00, 0xc0000386f8, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1494 +0x4b
github.com/lightningnetwork/lnd/routing.(*paymentLifecycle).createNewPaymentAttempt(0xc0085c4f00, 0x40, 0x40, 0xc001034b80, 0xc01281b278, 0x52e342)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:262 +0x475
github.com/lightningnetwork/lnd/routing.(*paymentLifecycle).resumePayment(0xc0085c4f00, 0x0, 0x0, 0x0, 0x0, 0x1, 0xb4e4ac, 0xc000010120)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/payment_lifecycle.go:52 +0xd0a
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).sendPayment(0xc0004d0140, 0x0, 0xc0097def00, 0x14d6860, 0xc007ab2b40, 0x0, 0x0, 0x0, 0x0, 0x1151b00, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1812 +0x1d7
github.com/lightningnetwork/lnd/routing.(*ChannelRouter).SendToRoute(0xc0004d0140, 0x58ca5384bd9010e2, 0xc79742f1db16e50, 0x64bf8000ad875178, 0x7e996cf4e75018da, 0xc00648a140, 0x0, 0x0, 0x0, 0x0, ...)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/routing/router.go:1724 +0x20f
github.com/lightningnetwork/lnd.(*rpcServer).dispatchPaymentIntent(0xc0001f8160, 0xc018f88420, 0x0, 0x0, 0x0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3172 +0xac
github.com/lightningnetwork/lnd.(*rpcServer).sendPaymentSync(0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc01fefe200, 0x0, 0xc005b21700, 0x1514d40)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3412 +0x11f
github.com/lightningnetwork/lnd.(*rpcServer).SendToRouteSync(0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc00648a050, 0xc0001f8160, 0xd26625, 0xc001652360)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/rpcserver.go:3387 +0x9c
github.com/lightningnetwork/lnd/lnrpc._Lightning_SendToRouteSync_Handler.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x0, 0x0, 0x10, 0xc001898000)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:11745 +0x89
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x1199ba0, 0xc015510001, 0xc00dd82180, 0xc005b21920)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.0/chain.go:31 +0x114
github.com/lightningnetwork/lnd.errorLogUnaryServerInterceptor.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00648a0a0, 0xc00fa51ba0, 0xc01c86c010, 0xc015510000, 0xc00648a0a0)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/log.go:248 +0x81
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0x1, 0x1, 0x0, 0x0)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.0/chain.go:34 +0x9c
github.com/lightningnetwork/lnd/macaroons.(*Service).UnaryServerInterceptor.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00648a0a0, 0x0, 0xc001c04100, 0xc005b21a10, 0x40e6a8)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/macaroons/service.go:124 +0x150
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1(0x14f5c40, 0xc007ab2750, 0x115eba0, 0xc00648a050, 0xc00fa51b40, 0xc00fa51b60, 0xc0117a3a80, 0x5aa858, 0x11344e0, 0xc007ab2750)
        /home/bitcoin/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.0/chain.go:39 +0x155
github.com/lightningnetwork/lnd/lnrpc._Lightning_SendToRouteSync_Handler(0x11eec80, 0xc0001f8160, 0x14f5c40, 0xc007ab2750, 0xc00648a000, 0xc0014dc960, 0x14f5c40, 0xc007ab2750, 0xc01001a000, 0x85d)
        /home/bitcoin/go/src/github.com/lightningnetwork/lnd/lnrpc/rpc.pb.go:11747 +0x14b
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100, 0xc0014dcd50, 0x1d80960, 0x0, 0x0, 0x0)
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:966 +0x46a
google.golang.org/grpc.(*Server).handleStream(0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100, 0x0)
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:1245 +0xd16
google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00abd0d70, 0xc000b4c000, 0x1504140, 0xc009372000, 0xc001c04100)
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:685 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /home/bitcoin/go/pkg/mod/google.golang.org/grpc@v1.18.0/server.go:683 +0xa1
@LNBIG-COM
Copy link
Author

After restart i have many like these messages. Is it very bad? Possible loss of funds in this case? Or will everything come back with time?

2019-11-12 13:07:20.640 [INF] LNWL: Done catching up block hashes
2019-11-12 13:07:20.640 [INF] LNWL: Finished rescan for 212915 addresses (synced to block 0000000000000000000e16bedab7553cdbf4fcb007c87ab2f9eec9acec66f0de, height 603458)
2019-11-12 13:07:21.537 [INF] LTND: Chain backend is fully synced (end_height=603458)!
2019-11-12 13:07:21.542 [INF] PRNF: PeerNotifier starting
2019-11-12 13:07:21.617 [INF] HSWC: Starting HTLC Switch
2019-11-12 13:07:21.617 [INF] NTFN: New block epoch subscription
2019-11-12 13:07:21.617 [INF] NTFN: New block epoch subscription
2019-11-12 13:07:22.333 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574243:574:0, HTLC ID = 27
2019-11-12 13:07:22.333 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574243:574:0, HTLC ID = 27
2019-11-12 13:07:22.485 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:672:1, HTLC ID = 416
2019-11-12 13:07:22.485 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:672:1, HTLC ID = 416
2019-11-12 13:07:22.497 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 566066:1882:0, HTLC ID = 80
2019-11-12 13:07:22.497 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 566066:1882:0, HTLC ID = 80
2019-11-12 13:07:22.503 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2702:0, HTLC ID = 107
2019-11-12 13:07:22.512 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2702:0, HTLC ID = 107
2019-11-12 13:07:22.989 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550423:10:0, HTLC ID = 396
2019-11-12 13:07:22.989 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550423:10:0, HTLC ID = 396
2019-11-12 13:07:23.019 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 586988:500:1, HTLC ID = 49
2019-11-12 13:07:23.019 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 586988:500:1, HTLC ID = 49
2019-11-12 13:07:24.423 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2728:0, HTLC ID = 11
2019-11-12 13:07:24.423 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 565660:2728:0, HTLC ID = 11
2019-11-12 13:07:25.521 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 601227:586:0, HTLC ID = 0
2019-11-12 13:07:25.521 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 601227:586:0, HTLC ID = 0
2019-11-12 13:07:25.564 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 108
2019-11-12 13:07:25.564 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 182
2019-11-12 13:07:25.564 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 108
2019-11-12 13:07:25.564 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:1362:0, HTLC ID = 182
2019-11-12 13:07:26.403 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 586988:489:1, HTLC ID = 155
2019-11-12 13:07:26.403 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 586988:489:1, HTLC ID = 155
2019-11-12 13:07:26.439 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550739:2187:0, HTLC ID = 382
2019-11-12 13:07:26.439 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550739:2187:0, HTLC ID = 382
2019-11-12 13:07:26.469 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567233:1602:0, HTLC ID = 59
2019-11-12 13:07:26.469 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567233:1602:0, HTLC ID = 59
2019-11-12 13:07:27.267 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 141
2019-11-12 13:07:27.267 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 142
2019-11-12 13:07:27.267 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 141
2019-11-12 13:07:27.267 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 142
2019-11-12 13:07:27.267 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 175
2019-11-12 13:07:27.267 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2279:0, HTLC ID = 175
2019-11-12 13:07:27.299 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:669:1, HTLC ID = 123
2019-11-12 13:07:27.299 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:669:1, HTLC ID = 123
2019-11-12 13:07:27.328 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567206:952:1, HTLC ID = 145
2019-11-12 13:07:27.328 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567206:952:1, HTLC ID = 145
2019-11-12 13:07:28.909 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2284:0, HTLC ID = 237
2019-11-12 13:07:28.909 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567217:2284:0, HTLC ID = 237
2019-11-12 13:07:28.928 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 70
2019-11-12 13:07:28.928 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 81
2019-11-12 13:07:28.928 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 70
2019-11-12 13:07:28.928 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2096:1, HTLC ID = 81
2019-11-12 13:07:28.965 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550333:1519:0, HTLC ID = 33
2019-11-12 13:07:28.965 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550333:1519:0, HTLC ID = 33
2019-11-12 13:07:28.997 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 13
2019-11-12 13:07:28.997 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 14
2019-11-12 13:07:28.997 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 13
2019-11-12 13:07:28.997 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 601227:584:0, HTLC ID = 14
2019-11-12 13:07:29.212 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 550330:685:0, HTLC ID = 220
2019-11-12 13:07:29.212 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 550330:685:0, HTLC ID = 220
2019-11-12 13:07:29.285 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 30
2019-11-12 13:07:29.285 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 38
2019-11-12 13:07:29.285 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 30
2019-11-12 13:07:29.285 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 38
2019-11-12 13:07:29.285 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 45
2019-11-12 13:07:29.285 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 574084:2130:1, HTLC ID = 45
2019-11-12 13:07:29.665 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 558535:1606:1, HTLC ID = 21
2019-11-12 13:07:29.665 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 558535:1606:1, HTLC ID = 21
2019-11-12 13:07:30.105 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 88
2019-11-12 13:07:30.105 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 94
2019-11-12 13:07:30.105 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 88
2019-11-12 13:07:30.105 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 579354:1194:1, HTLC ID = 94
2019-11-12 13:07:31.389 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 210
2019-11-12 13:07:31.389 [ERR] HSWC: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 293
2019-11-12 13:07:31.389 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 210
2019-11-12 13:07:31.389 [ERR] HSWC: unhandled error while reforwarding htlc settle/fail over htlcswitch: Unable to find target channel for HTLC settle/fail: channel ID = 567234:673:1, HTLC ID = 293

@wpaulino
Copy link
Contributor

wpaulino commented Nov 12, 2019

panic: runtime error: index out of range [20] with length 20

Was a route with more than 20 hops provided?

After restart i have many like these messages. Is it very bad? Possible loss of funds in this case? Or will everything come back with time?

They can be ignored. We had a recent change that caused these to be logged without it actually being a fatal error, but we've yet to address that (see #3656).

@joostjager
Copy link
Contributor

Yes, sounds like max hops exceeded. We need to fix that. From the code

// NumMaxHops is the maximum path length. This should be set to an estimate of
// the upper limit of the diameter of the node graph.
//
// TODO(roasbeef): adjust due to var-payloads?
const NumMaxHops = 20

@LNBIG-COM
Copy link
Author

LNBIG-COM commented Nov 13, 2019

panic: runtime error: index out of range [20] with length 20

Was a route with more than 20 hops provided?

I no longer have detailed logs from my scripts about this incident. But may be...
But if this is so, this is not a reason to crash ;-)

@joostjager
Copy link
Contributor

Do you append hops to the result from QueryRoutes? Because the pure QueryRoutes result should never be longer than 20 hops. I think that you append your own node to complete the circular route?

Still no reason to crash :)

@LNBIG-COM
Copy link
Author

LNBIG-COM commented Nov 13, 2019

Do you append hops to the result from QueryRoutes?

No

Because the pure QueryRoutes result should never be longer than 20 hops. I think that you append your own node to complete the circular route?

Still no reason to crash :)

After fail payment i add one failed direct way (parsed from error of SendToRouteSync - ShortChannelId) of edge to ignored_pairs but if one node does more fails i add this node to ignored_nodes. And the ignore_pairs consists before start all channels from LNBIG (excluding two channels that need to be rebalanced). Due to these big lists i think yes, it happens. I don't have other way to find paths from one LNBIG channel to other LNBIG channel through outside.

I think this bug should be fixed otherwise every time of rebalancing i will repair LND and write here new logs ;-)

@LNBIG-COM
Copy link
Author

The code there i caught this crash is here (NodeJS):

https://gist.github.com/LNBIG-COM/90f143c8ba61b848eb33dc4253a90425

@joostjager
Copy link
Contributor

I think you do append in that code?
route.hops.push

Side note: I see your recalculateRoute. An alternative for that is to use the new BuildRoute rpc

@LNBIG-COM
Copy link
Author

LNBIG-COM commented Nov 13, 2019

I think you do append in that code?
route.hops.push

First, I do not calculate the complete route, but to the N-1 node, then I add the last hop (receiver). Without this, it is impossible to calculate a route for rebalancing - for example, when node A wants to find a route to node A (but to a different channel) - without this lnd will say that there are no routes. I wrote about this here: #3689

@LNBIG-COM
Copy link
Author

LNBIG-COM commented Nov 13, 2019

An alternative for that is to use the new BuildRoute rpc

May be. I don't know about this. This function is missed in lnd-0.8.0-beta

@joostjager
Copy link
Contributor

Yes, I understand that that is necessary. But I also think that pushes the route length to 21 hops. But issue is clear, I will think about a solution. There is also no way currently to limit the max route length with queryroutes. Otherwise you could set it to 19 and still be able to add the last hop manually.

@LNBIG-COM
Copy link
Author

LNBIG-COM commented Nov 13, 2019

But I also think that pushes the route length to 21 hops.

Ohh, may be. Thanks for this info. I didn't think about this ;-)

Otherwise you could set it to 19 and still be able to add the last hop manually.

I will do! Thanks!

@joostjager
Copy link
Contributor

Currently, you unfortunately cannot set the max number of hops on a queryroutes call.

But at least you can check whether the route returned from queryroutes is 20 hops. If that is the case, don't fire SendToRoute if you don't want to crash the node

@LNBIG-COM
Copy link
Author

The limitation is already there and while the flight is normal :)

@joostjager
Copy link
Contributor

Related issue: #3725

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Unintended code behaviour multi-hop onion routing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants