Bridge daemon exited when a withdrawn transaction fails

[sameh-farouk]

Describe the bug

The bridge daemon can start even if the bridge account TFT balance is zero, without any alert or warning, then as soon as it tries to process a withdrawn transaction, it stops completely (call os.Exit(1)).

In my opinion, the program should check for low and empty balances at startup and predictively. Additionally, any transaction errors should be handled properly without causing the program to exit.

To Reproduce

Steps to reproduce the behavior:

1. Start the bridge with a zero TFT balance account.
2. Initiate a withdraw flow using the swap_to_stellar extrinsic.
3. Wait a few seconds until the event is emitted from TFchain and read by the bridge daemon.
4. The daemon will exit with fatal error.

Expected

• You should receive an error when starting the bridge with an empty wallet.
• You should see a warning when a low or empty balance is detected during the bridge operation.
• Any transaction errors should be handled and logged without affecting the bridge stability or availability.

Screenshots

logs from the bridge

3:13PM INF found burn transaction ready event ID=1
3:13PM INF fetching events for blockheight ID=97
3:13PM INF fetching stellar transactions account=GBD3PXJEQOCQ5VR2JSMFNXYBBQF5RDEZP5GMTXDYZWMNZQJHR6HFX3AJ cursor=9185028080668672 horizon=https://horizon-testnet.stellar.org/
3:13PM INF horizon error: "Transaction Failed" (tx_failed, op_underfunded) - check horizon.Error.Problem for more information
3:13PM ERR error while submitting transaction map[envelope_xdr:AAAAAgAAAABHt90kg4UO1jpMmFbfAQwL2IyZf0zJ3HjNmNzBJ4+OWwABhqAAIKG7AAAAAwAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAABAAAAAEe33SSDhQ7WOkyYVt8BDAvYjJl/TMnceM2Y3MEnj45bAAAAAQAAAAD4yhflnZxmxIZTIWFtrkmC0EiFaR1EzW23Cjc2vREpMwAAAAFURlQAAAAAADn8ZBt6i0yx64bD0uibElAe/mY1PyeKE9WATCm/959kAAAAAAVdSoAAAAAAAAAAASePjlsAAABAemsDNBz402BviP0pRMSefDsHTLmflqJ/jDDhF70yX1x/RCUIigolqSrDdgU12eILEYj5BtKaoxDwTi3ki0bgCw== result_codes:map[operations:[op_underfunded] transaction:tx_failed] result_xdr:AAAAAAAAAGT/////AAAAAQAAAAAAAAAB/////gAAAAA=] error="horizon error: \"Transaction Failed\" (tx_failed, op_underfunded) - check horizon.Error.Problem for more information"
3:13PM INF resetting account sequence
3:13PM FTL exited unexpectedly error="failed to handle withdraw ready: error submitting transaction: horizon error: \"Transaction Failed\" (tx_failed, op_underfunded) - check horizon.Error.Problem for more information"

[sameh-farouk]

Update:
This recently merged PR (#881) adds a feature to the bridge that reports the wallet balance every minute so it is easy to set alerts via some tools like loki. It also do a pre-check on wallet balance on startup to ensures that the bridge won't starts with an empty wallet (a wrong wallet maybe).

However, there is a potential issue that needs to be addressed: what should the bridge do when it receives a withdraw request that exceeds its stellar wallet balance? I would appreciate some feedback on two main aspects of this problem.

First, the current behavior is that the bridge daemon stop when this happens, which is undesirable because even the bridge wallet don't have enough fund to handle this specific (possibly large) transfer it affects the availability of the service and prevents it from handling other smaller transfers. I propose to change this behavior to one of the following options:

• Option 1: Log a warning or an error and skip the transfer. The transfer will be retried periodically until the wallet has enough funds to execute it. This option avoids refunding the user, but it may cause some transfers to be stuck for a long time. an alert can be configured as well for such transfers.
• Option 2: Refund the transfer (minus bridge fees) and cancel it. This option ensures that no transfer is stuck, but it may upset the user who has to pay the fees without getting the service.

Second, to reduce the likelihood of this issue occurring, I have some suggestions (but none of them is perfect):

• Suggestion 1: The bridge frontend could validate the bridge wallet balance and prevent the user from submitting a transfer request that is greater than the wallet balance. This suggestion may not work in all cases, as the balance may change due to concurrent requests or network delays.
• Suggestion 2: The bridge could impose a maximum limit on the transfer amount per transaction (both in the frontend and the backend). This suggestion may help to set up a reasonable wallet threshold, as it prevents a single transaction from draining the wallet. However, it may also limit the functionality of the bridge for some users who want to transfer large amounts.

I am not sure if any of these suggestions are already implemented in the frontend.
Please let me know your thoughts on this issue. Thank you.

[muhamadazmy]

Can we have Option 1 with Suggestion 1.
The UI can give best effort of how much transfer you can make. So basically the transfer is limited to say 80% of the total amount you have on the wallet.

At the same time, duo to delay in UI where the user can submit a transfer that is still more than the available balance, we still can put it a side, trigger a warning for operators to fund the wallet and then try again when there is enough balance.

[sameh-farouk]

Update:
Kinda blocked today testing the bridge changes as no TFT issuer ATM on stellar testnet. Blocked on https://github.com/threefoldtech/tf_operations/issues/2105

[sameh-farouk]

Verified
Now we Log the error and skip the transfer. The transfer will be retried periodically until the wallet has enough funds to execute it.