Bugfix/deadlock on adjust limits

[shashi165]

Summary:
Here is the root cause of the issue

1. We see database deadlock issue when there are 2 simultaneous requests to update NDC.
2. The central-ledger service receives these 2 requests and tries to update the NDC for both of them at the same time. Here is what happens at the database level.
   a. first request comes in and it gets a lock on the row before updating it

select * from `participantLimit` where `participantCurrencyId` = 65 and `participantLimitTypeId` = 1 and `isActive` = 1 for update;

b. second request comes in and tries to get a lock on another row of the same table

select * from `participantLimit` where `participantCurrencyId` = 59 and `participantLimitTypeId` = 1 and `isActive` = 1 for update;

3. Both of these statements have for update clause. It will cause the row to be locked to avoid dirty read (When a transaction is allowed to read a row that has been modified by an another transaction which is not committed yet)
   

   central-ledger/src/models/participant/facade.js

   Lines 490 to 495 in c972dfb

   	const existingLimit = await knex('participantLimit').transacting(trx).forUpdate().select('*')
   	.where({
   	participantCurrencyId: participantCurrencyId,
   	participantLimitTypeId: limitType.participantLimitTypeId,
   	isActive: 1
   	})

4. The way mysql select for update works, it will lock the row when there is an unique index on the columns in the where clause of the query. If there is no index on the where clause columns, it will lock the entire table.
5. In our query the where clause columns are
   participantCurrencyId, participantLimitTypeId and isActive
   And there is no unique index covering these columns on participantLimit table.
6. So instead of locking a particular row, each of the query above tries to lock the entire table and that causes the dead lock on the database, so database decides to kill on of the transaction to resolve the deadlock. We get following error from central-ledger

FSPIOPError: select * from `participantLimit` where `participantCurrencyId` = 65 and `participantLimitTypeId` = 1 and `isActive` = 1 for update - ER_LOCK_DEADLOCK: Deadlock found when trying to get lock; try restarting transaction

7. Sometimes due to the delay in the network(or other conditions), one of these requests is received before the other and we do not have the deadlock on the database. This explains why we see the issue intermittently.

Solution

Create an unique index on the participantLimit table which covers all the columns in the where clause.
participantCurrencyId, participantLimitTypeId, isActive
we also need to add participantLimitId column to make that index unique.

We might see similar issues for other API endpoints, if they meet the above conditions.

Also added some logging, it was impossible to debug the issue due to lack of logging. We should take a relook at all the logging.

[partiallyordered]

[mdebarros]

+1

Great find!

Worth noting that we have not experienced this specific issue in our performance tests. I am interested in seeing what the impact of this change will have on the performance characteristics.

Also, are you able to share your Kafka Client configuration for the Position Handler? I wonder if there is some config difference that is causing this issue to materialize on your side, and not in the OSS performance environment.

[shashi165]

+1

Great find!

Worth noting that we have not experienced this specific issue in our performance tests. I am interested in seeing what the impact of this change will have on the performance characteristics.

Also, are you able to share your Kafka Client configuration for the Position Handler? I wonder if there is some config difference that is causing this issue to materialize on your side, and not in the OSS performance environment.

@mdebarros , we have managed kafka in Mowali UAT env, that's where we discovered the issue. Apart from that we use the default configuration that we get from mojaloop/helm