Unable to pay Kraken deposit invoices

[noiamyourfather]

Issue and Steps to Reproduce

I get the following error when trying to pay an invoice (Kraken deposit invoice).
The following error is displayed almost immediately:

INFO    plugin-pay: cmd 45 partid 0: Destination 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69 is not reachable directly and all routehints were unusable.

Doing a lightning-cli getroute 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69 10000000 1 works as expected. The route is displayed correctly (3 hops in my case).
I have a single channel open with a single peer, but I've confirmed I can pay invoices generated by at least 2 different lightning wallets, and that the peer I have a channel open with has more than 100 other open channels.

I know of at least one other similar report of a c-lightning user trying to pay a kraken deposit invoice and seeing the very same message.

Some more info:

lightning-cli listnodes | grep nodeid | wc -l
 18276

I also tried connecting to their node directly, but no luck.

[PierreRochard]

Please let us know if there's anything we can do to help from the Kraken side 🙏

[instagibbs]

seems related to #4404

[instagibbs]

Any other nearby logs that look like "Removed routehint" or related?

Can we get a sample invoice from Kraken @PierreRochard

[beefgroin]

I have exactly the same issue, very annoying. I can transfer funds to my Phoenix wallet with no issues and then pay Kraken invoice, but not directly. And the route exists for sure.

{
   "code": 210,
   "message": "Destination 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69 is not reachable directly and all routehints were unusable.",
   "attempts": [
      {
         "status": "failed",
         "failreason": "Destination 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69 is not reachable directly and all routehints were unusable.",
         "partid": 0,
         "amount": "100000000msat"
      }
   ]
}

[mrtestyboy781]

@instagibbs

lnbc10u1p39ldhgpp5usmakqzvj3d83ykn00qv9ryfavrtwjj82lx2g6a6edt392z5h6aqdqhfdexz6m9dcsygetsdaekjaqcqzysxqyd9uqrzjqtlrha6clp7ekr8wd0d055ksgegd465vfrm30q6ckyctvtsn93h47ze2f5qquegqqsqqqqqqqqqqpjqq9qsp564lc5l8d2djw83t6keyskf9cm4fc4ptunyh53ztnsuw8eh4crpxs9qyyssqaxkq4l08sqluvzuyadtfqgdjc3dqz70dnzw02l2n6fw3vc79lfyzxx2flyhucdwm53mpvvyjv7usw3ng4wwwzv5v9cl299k6uuv0wcspkdg3a8

Is open for 5 days, let me know if you need another. Anyone reading, please don't actually pay it

[rustyrussell]

Hmm, does "paystatus" give any clues? That sometimes has a more detailed log of what happened for a payment...

[mrtestyboy781]

@beefgroin - are you able to reproduce this? could you possibly try the paystatus command as rusty suggested? On my c-lightning node i have never hit this issue.

[beefgroin]

@beefgroin - are you able to reproduce this? could you possibly try the paystatus command as rusty suggested? On my c-lightning node i have never hit this issue.

hey, yes, I'm still not able to pay Kraken's invoice. Just tried. Spark wallet says:
Ran out of routes to try after 106 attempts: see paystatus

And then when I paystatus I see

bolt11: >-
      lnbc1m1p3xz5sfpp53egssl0940kd9zp9smz0x724glahe6tvta34m0epa3zrjyx0lv5qdqhfdexz6m9dcsygetsdaekjaqcqzysxqzjcrzjqtlrha6clp7ekr8wd0d055ksgegd465vfrm30q6ckyctvtsn93h47ze2f5qquegqqsqqqqqqqqqqpjqq9qsp5sp4s62f2hym3takykfrhyxtm4vzmh3dsz778y08mv008eeaw749s9qyyssq29m2lych0ludeplvweyczrn0s4u8t7safqf2qzqrhv28z5yft368eysxztuzhtefqu3rgf4naepz7kgh600h892jqnmvq6pn5f35c0qpj4wt6s
    amount_msat: 100000000msat
    destination: 02f1a8c87607f415c8f22c00593002775941dea48869ce23096af27b0cfdcc0b69
    attempts:
      - strategy: >-
          Split into 11 sub-payments due to initial size (100000000msat >
          10000000msat)
        start_time: '2022-04-21T12:46:26.877Z'
        age_in_seconds: 262
        end_time: '2022-04-21T12:46:28.679Z'
        state: completed
      - start_time: '2022-04-21T12:46:28.679Z'
        age_in_seconds: 260
        end_time: '2022-04-21T12:46:35.580Z'
        state: completed
        failure:
          message: 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)'
          code: 204
          data:
            id: 52
            raw_message: >-
              10070088befa6344257c86b2333f246f2a944a9c2eb70670fdfad9277c2a84f37c841a124e16f0c9b2789c4bf20359c22e69523b7d0fe1354edbc1b72a392e2a012e27ee6fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d61900000000000af3520002bf00016260b802010000c800000000000003e8000000640000006400000001270b0180
            failcode: 4103
            failcodename: WIRE_TEMPORARY_CHANNEL_FAILURE
            erring_index: 1
            erring_node: 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f
            erring_channel: 717650x703x1
            erring_direction: 0
      - strategy: Still have 9 attempts left
        start_time: '2022-04-21T12:46:35.580Z'
        age_in_seconds: 253
        end_time: '2022-04-21T12:46:39.518Z'
        state: completed
        failure:
          message: 'failed: WIRE_TEMPORARY_CHANNEL_FAILURE (reply from remote)'
          code: 204
          data:
            id: 65
            raw_message: >-
              10070088cd581bf63e310f966c9937f9586c661b6699fdcb8957b2e6d505112a949f5e68645cc1eb69fb5b8ffce4e9ca313bd29898652d45facd3a7eedaed15b171456c36fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d61900000000000af4220009a7000062602b63010000c800000000000003e80000006400000064000000003b023380
            failcode: 4103
            failcodename: WIRE_TEMPORARY_CHANNEL_FAILURE
            erring_index: 1
            erring_node: 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f
            erring_channel: 717858x2471x0
            erring_direction: 0

And then million times the variation of the last part apparently with different nodes.

[beefgroin]

I think there was something wrong with my node, cause I couldn't even pay any other invoices. After I've restarted my Raspiblitz, I was finally able to deposit to Kraken, I guess they've fixed it on their side :)

[vincenzopalazzo]

I think there was something wrong with my node, cause I couldn't even pay any other invoices. After I've restarted my Raspiblitz, I was finally able to deposit to Kraken, I guess they've fixed it on their side :)

Maybe you got only a good channel choice during the pathfinding! but yes, these is somethings that we should look inside

[rustyrussell]

Yes, this pattern of failures you see is very much "nobody has liquidity". We should implement a WTF plugin which shows why a given payment would/may fail. Anyone want to try implementing it?

[vincenzopalazzo]

We should implement a WTF plugin which shows why a given payment would/may fail. Anyone want to try implementing it?

This is one of my summer of bitcoin projects proposal with a guy that will work on a plugin system with dart, so we can compile the plugin in a binary! The idea of the plugin is also to include other types of pay algorithm