Pay plugin crashes when trying to pay Breez invoice

[hosiawak]

Issue and Steps to Reproduce

Here's the Breez invoice:

lnbc34320n1psf7ng9pp5ptt4wa32mwr3047gxytsdh5u6e4maxufulj7zypwdfx4hcwxzkvsdq2vfex2et6xgcqzpgxqrrssrzjqvgptfurj3528snx6e3dtwepafxw5fpzdymw9pj20jj09sunnqmwqqqqqyqqqqqqqqqqqqlgqqqqqqgqjqnp4qvtacy8q08fxngz0pyj26e4phmsvcf9n0qhnw2x5ldq0c58p9dkwqsp5qd5u7z72t3s8jzw2094xdda2aj3tmxrrrlh0fzyazjjalv88zgls9qy9qsqxw6x5r7ftqzxq3tk4pcn2503yqne5dqszp0vw9mewymqfzkrg78p8twphuya2dm632vzvk9z28uqs5nk5h98jfgkm860wm4ffhxsn8gpmlygh6

Running c-lightning 0.10.0 on FreeBSD 12.2-RELEASE-p6

Trying to pay the above invoice (or any other where the payee is private/unknown - eg. Phoenix or @lntxbot) results in a crash. Some debug info:

Assertion failed: (localoff + len <= tal_bytelen(map->local)), function map_copy, file common/gossmap.c, line 94.

And here's the output from lldb (FreeBSD equivalent of gdb) attached to the pay process:

(lldb) bt
* thread #1, name = 'pay', stop reason = signal SIGABRT
  * frame #0: 0x000000080082ec2a libc.so.7`__sys_thr_kill + 10
    frame #1: 0x000000080082d084 libc.so.7`__raise + 52
    frame #2: 0x00000008007a3279 libc.so.7`abort + 73
    frame #3: 0x000000080081d291 libc.so.7`__assert + 81
    frame #4: 0x00000000002a0040 pay`map_copy(map=0x0000000800af3028, offset=2931885032, dst=0x00007fffffffdf18, len=8) at gossmap.c:94:3
    frame #5: 0x000000000029e409 pay`map_be64(map=0x0000000800af3028, offset=2931885032) at gossmap.c:132:2
    frame #6: 0x000000000029e3d2 pay`gossmap_chan_scid(map=0x0000000800af3028, c=0x000000080348f708) at gossmap.c:533:13
    frame #7: 0x00000000002510a0 pay`route_hops_from_route(ctx=0x0000000800b68228, p=0x0000000800b68228, r=0x0000000800abad98) at libplugin-pay.c:702:24
    frame #8: 0x000000000024d96a pay`payment_getroute(p=0x0000000800b68228) at libplugin-pay.c:809:13
    frame #9: 0x000000000024d4db pay`payment_continue(p=0x0000000800b68228) at libplugin-pay.c:2012:4
    frame #10: 0x0000000000250c71 pay`adaptive_splitter_cb(d=0x0000000800ae4658, p=0x0000000800b68228) at libplugin-pay.c:3684:2
    frame #11: 0x000000000024d498 pay`payment_continue(p=0x0000000800b68228) at libplugin-pay.c:2004:10
    frame #12: 0x000000000024f061 pay`retry_step_cb(rd=0x0000000800ae4628, p=0x0000000800b68228) at libplugin-pay.c:2180:10
    frame #13: 0x000000000024d498 pay`payment_continue(p=0x0000000800b68228) at libplugin-pay.c:2004:10
    frame #14: 0x000000000024fdd4 pay`waitblockheight_cb(d=0x0000000000000000, p=0x0000000800b68228) at libplugin-pay.c:3252:10
    frame #15: 0x000000000024d498 pay`payment_continue(p=0x0000000800b68228) at libplugin-pay.c:2004:10
    frame #16: 0x000000000025024e pay`presplit_cb(d=0x0000000800ae45f8, p=0x0000000800b68228) at libplugin-pay.c:3477:11
    frame #17: 0x000000000024d498 pay`payment_continue(p=0x0000000800b68228) at libplugin-pay.c:2004:10
    frame #18: 0x0000000000257aba pay`payee_incoming_limit_count(cmd=0x0000000000000000, buf="\n\n{\"jsonrpc\":\"2.0\",\"id\":4,\"result\":{\"channels\":[]} }\n\n", result=0x0000000800af31a0, p=0x0000000800b68228) at libplugin-pay.c:3768:2
    frame #19: 0x000000000024c08f pay`handle_rpc_reply(plugin=0x0000000800ae7028, toks=0x0000000800af3128) at libplugin.c:564:9
    frame #20: 0x000000000024bda4 pay`rpc_read_response_one(plugin=0x0000000800ae7028) at libplugin.c:711:2
    frame #21: 0x000000000024bb11 pay`rpc_conn_read_response(conn=0x0000000800b031a8, plugin=0x0000000800ae7028) at libplugin.c:731:9
    frame #22: 0x00000000002875c6 pay`next_plan(conn=0x0000000800b031a8, plan=0x0000000800b031c8) at io.c:59:9
    frame #23: 0x000000000028833e pay`do_plan(conn=0x0000000800b031a8, plan=0x0000000800b031c8, idle_on_epipe=false) at io.c:407:10
    frame #24: 0x00000000002881a6 pay`io_ready(conn=0x0000000800b031a8, pollflags=1) at io.c:417:8
    frame #25: 0x000000000028a2e9 pay`io_loop(timers=0x0000000800ae7130, expired=0x00007fffffffe9f8) at poll.c:445:5
    frame #26: 0x0000000000248a2c pay`plugin_main(argv=0x00007fffffffeb70, init=(pay`init at pay.c:1912), restartability=PLUGIN_RESTARTABLE, init_rpc=true, features=0x0000000000000000, commands=0x000000000020f490, num_commands=4, notif_subs=0x0000000000000000, num_notif_subs=0, hook_subs=0x0000000000000000, num_hook_subs=0) at libplugin.c:1414:3
    frame #27: 0x000000000023e8c6 pay`main(argc=1, argv=0x00007fffffffeb70) at pay.c:2199:2
    frame #28: 0x000000000023e280 pay`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7 

[hosiawak]

Actually, it's any invoice now, not just the ones mentioned above. I'll try removing gossip_store and see what happens.

[hosiawak]

Removing gossip_store fixed it. I can pay any invoice now.

[hosiawak]

Actually, I managed to pay the original Breez invoice but when I attempted to pay the following @lntxbot one it crashed with the same error and now is crashing when attempting to pay any invoice:

lnbc23450n1psf70tnpp5w7l22dkxta272gle0y7cunyt6rhsxacgmxql2lrupqft5faerp8qdqgw3jhxap4xq9p5hsqrzjqtqkejjy2c44jrwj08y5ygqtmn8af7vscwnflttzpsgw7tuz9r40lccjk9rvhqg9dvqqqqd3qqqq89gqjqsp5qypqxpq9qcrsszg2pvxq6rs0zqg3yyc5z5tpwxqergd3c8g7rusq9qypqsqedv98nrpxhrs9vspf9kgyg08xd957ydwjy4359ylem750tqzz4vngyyfjsq4ya3x8p3jd54j3sfq92y9fu2yg0dsnhwa8z2j60504jcqh76j38

[hosiawak]

I installed the latest master 0f2009c and now getting a different assertion error:

Assertion failed: (n < node->num_chans), function gossmap_nth_chan, file common/gossmap.c, line 951.

[hosiawak]

I deleted gossip_store then managed to pay 1 @lntxbot invoice:

lnbc3440n1psf7l96pp5laun4wjuaygc5q4yemd2rtfhuygfmkzhexztz0nvdz67xepetfhqdq2w3jhxap3xqxq9p5hsqrzjqtqkejjy2c44jrwj08y5ygqtmn8af7vscwnflttzpsgw7tuz9r40lz547t4w7ydy9sqqqqd3qqqq89gqjqsp5qypqxpq9qcrsszg2pvxq6rs0zqg3yyc5z5tpwxqergd3c8g7rusq9qypqsqfgj2re0vmyegp28wykrxuytqww5vcd8vadcjcyyjxxsrr0lact3p37930au7maf6undl9dng6ftr5u064qzutfs2pkpyfwle0u4xxuqqlex68t

but subsequent attempts at paying eg.:

lnbc34450n1psf7lkvpp5zahhgqvq0pdmqra3hnsf6s4h53vflff69w0r38fchlpyd6wlf9ksdq2w3jhxap3xyxq9p5hsqrzjqtqkejjy2c44jrwj08y5ygqtmn8af7vscwnflttzpsgw7tuz9r407dl9sn0r48klxsqqqqd3qqqq89gqjqsp5qypqxpq9qcrsszg2pvxq6rs0zqg3yyc5z5tpwxqergd3c8g7rusq9qypqsqeyg32ug72m64k5pgzjmvyk28s5zc5cyayz24uva0jxfpjhvxxa6sqtpqqgxaj76x9hxca27mrfryvnclt6hujlls9hgcc6dnl6ty7ccpnggj8p

result in the original assertion error:

Assertion failed: (localoff + len <= tal_bytelen(map->local)), function map_copy, file common/gossmap.c, line 94.

[hosiawak]

Now trying to pay a 200sat Blue Wallet invoice:

lnbc2u1ps2ygr2pp5vaq5tp3lw9lgmmrkke034f6xx3a80pz9qgnparn75yrvvewde0usdqqcqzpgxqyz5vqsp5ty4xl2s86nzepmspyz4ufk4gce4mjv0a0hydf7gkdej0e4vdvafq9qyyssqg5uq9gylymukt7dgr4yzf8d5y8fz3mlaptc9fawar3e8mxzatfx90xphru4x9furgsl4jvuwyf6zvr7duhnclyfw3gur4t2u3k2l8xspt48zm3

Backtrace:

* thread #1, name = 'pay', stop reason = signal SIGABRT
    frame #0: 0x0000000800832c2a libc.so.7`__sys_thr_kill + 10
    frame #1: 0x0000000800831084 libc.so.7`__raise + 52
    frame #2: 0x00000008007a7279 libc.so.7`abort + 73
    frame #3: 0x0000000800821291 libc.so.7`__assert + 81
    frame #4: 0x00000000002a32f0 pay`map_copy(map=0x0000000800af7128, offset=2767126529, dst=0x00007fffffffded8, len=8) at gossmap.c:94:3
    frame #5: 0x00000000002a16b9 pay`map_be64(map=0x0000000800af7128, offset=2767126529) at gossmap.c:132:2
    frame #6: 0x00000000002a1682 pay`gossmap_chan_scid(map=0x0000000800af7128, c=0x0000000803485b50) at gossmap.c:533:13
    frame #7: 0x0000000000251a50 pay`route_hops_from_route(ctx=0x0000000800b62028, p=0x0000000800b62028, r=0x0000000800ac1ae8) at libplugin-pay.c:702:24
  * frame #8: 0x000000000024e26a pay`payment_getroute(p=0x0000000800b62028) at libplugin-pay.c:809:13

(lldb) v
(payment *) p = 0x0000000800b62028
(const dijkstra *) dij = 0x0000000800b645a8
(const gossmap_node *) dst = 0x0000000801355018
(const gossmap_node *) src = 0x0000000801341428
(route **) r = 0x0000000800ac1ae8
(amount_msat) fee = (millisatoshis = 34371358720)
(gossmap *) gossmap = 0x0000000800af7128
(bool (*)(const gossmap *, const gossmap_chan *, int, amount_msat, payment *)) can_carry = 0x0000000000251890 (pay`payment_route_can_carry at libplugin-pay.c:671)

(lldb) p *src
(const gossmap_node) $50 = {
  nann_off = 629
  num_chans = 74
  chan_idxs = 0x0000000800ac3640
}
(lldb) p *dst
(const gossmap_node) $51 = {
  nann_off = 14436492
  num_chans = 5
  chan_idxs = 0x000000080144e5c0
}

I don't know much about gossmap to navigate these offsets and figure out the actual node ids though. Please let me know what I can debug further if possible (pinging @cdecker )

[hosiawak]

Now I can't pay any type of invoice, still getting:

Assertion failed: (localoff + len <= tal_bytelen(map->local)), function map_copy, file common/gossmap.c, line 94.