Skip to content

Commit

Permalink
feat!: fx and interscheme implementation (#1058)
Browse files Browse the repository at this point in the history
* feat: added db migration scripts for fx

* feat(3574): update prepare-handler to deal with FX transfers (#988)

* feat(3574): updated prepare handler for FX

* feat(3574): added fxTransfer related tables;  updated validator for FX

* feat(3574): added fxTransfer related tables;  updated validator for FX

* feat(3574): comments/todos

* feat(3574): added fx roleTypes; added content.context

* feat(3574): added cyril; updated unit-tests

* feat(3574): added FX endpoints to seeds (#989)

* feat: implement changes in position handler for FX (#986)

* chore: updated central service shared lib

* feat: added some changes for fx flow

* feat: added changes for position prepare handler

* chore: refactor cyril functions

* feat: position-commit working

* feat: upgraded central shared

* chore(snapshot): 17.4.0-snapshot.0

* chore(snapshot): 17.4.0-snapshot.1

* chore(snapshot): 17.4.0-snapshot.2

* chore(snapshot): 17.4.0-snapshot.3

* chore(snapshot): 17.4.0-snapshot.4

* chore(snapshot): 17.4.0-snapshot.5

* chore(snapshot): 17.4.0-snapshot.6

* chore(snapshot): 17.4.0-snapshot.7

* chore(snapshot): 17.4.0-snapshot.8

* fix: positions

* fix: disable unit tests for snapshot

* chore(snapshot): 17.4.0-snapshot.9

* chore: added docs

* chore: updated doc

* fix: normal fulfil

* fix: normal flow

* fix: updated fx diagram

* chore: dep update

* chore(snapshot): 17.4.0-snapshot.10

* feat(mojaloop/#3689): fx quotes changes (#995)

* feat: add FX quotes endpointType and kafka topics
* chore: upgrade cs-shared
* chore: fix audit

* chore(snapshot): 17.4.0-snapshot.12

* ci: disable unit tests and test coverage runs for snapshots

* chore(snapshot): 17.4.0-snapshot.13

* fix: manual changes from upstream commits

* chore(snapshot): 17.7.0-snapshot.0

* chore(mojaloop/#3820): fix current tests and merge in main  (#1000)

* fix: cluster performance testing issues (#996)

* test: fix disconnect errors (#998)

* chore(release): 17.6.1 [skip ci]

* chore: fix current tests

* boolean

* chore: add endpoints to test data

* fix endpoint import

* chore: improve validator coverage

* chore: move prepare tests into file to match src structure

---------

Co-authored-by: Kalin Krustev <kalin.krustev@gmail.com>
Co-authored-by: mojaloopci <info@mojaloop.io>

* test(mojaloop/#3819): harden fx prepare flow  (#1002)

* chore: more coverage

* coverage

* test(mojaloop/#3819): prepare handler testing (#1004)

* test(mojaloop/#3819): prepare handler testing

* dep

* audit

* reconcile to one file due to producer bug #3067

* address comments

* chore: standardise position prepare handler (#1005)

* feat: added fx-position-prepare capability to batch handler

* chore: reverted fx implementation in non batch mode

* chore: added unit tests

* chore: dep, audit and lint

---------

Co-authored-by: Kevin Leyow <kleyow@gmail.com>

* feat(mojaloop/#3844): added corner cases impl. for FX; added unit-tests (#1006)

* feat(mojaloop/#3844): added corner cases impl. for FX; added unit-tests

* feat(mojaloop/#3844): added corner cases impl. for FX; added unit-tests

* feat(mojaloop/#3844): added fxTransferErrorDuplicateCheck table; moved fxFulfilt tests in a separare file

* feat(mojaloop/#3844): run tests with output

* feat(mojaloop/#3844): fixed unit-test on ci env

* feat(mojaloop/#3844): added unit-tests for FxFulfilService;  moved duplicateCheckComparator logic to service

* feat(mojaloop/#3844): reverted ci test-coverage

* feat(mojaloop/#3844): added license

* feat(mojaloop/#3844): moved checkErrorPayload to helpers

* feat(mojaloop/#3844): moved checkErrorPayload to helpers

* feat(mojaloop/#3844): updated from feat/fx-impl

* fix: removed fx position prepare integration tests in non batch mode (#1010)

* chore: fix int tests, lint and update deps (#1013)

* chore: lint and update deps

* int test

* chore: removed unneeded kafkaHelper; excluded some files from test-coverage check (#1015)

* feat(mojaloop/#3844): added integration tests for fxFulfil flow (#1011)

* feat(mojaloop/#3844): added corner cases impl. for FX; added unit-tests

* feat(mojaloop/#3844): added corner cases impl. for FX; added unit-tests

* feat(mojaloop/#3844): added fxTransferErrorDuplicateCheck table; moved fxFulfilt tests in a separare file

* feat(mojaloop/#3844): run tests with output

* feat(mojaloop/#3844): fixed unit-test on ci env

* feat(mojaloop/#3844): added unit-tests for FxFulfilService;  moved duplicateCheckComparator logic to service

* feat(mojaloop/#3844): reverted ci test-coverage

* feat(mojaloop/#3844): added license

* feat(mojaloop/#3844): moved checkErrorPayload to helpers

* feat(mojaloop/#3844): moved checkErrorPayload to helpers

* feat(mojaloop/#3844): updated from feat/fx-impl

* feat(mojaloop/#3844): added integration tests for fxFulfil flow

* feat(mojaloop/#3844): fixed  producer.disconnect() in int-tests

* feat(mojaloop/#3844): added test:int:transfers script

* feat(mojaloop/#3844): added duplicateCheck int test

* feat(mojaloop/#3844): small cleanup

* feat(mojaloop/#3844): added duplicate and fulfilment check int-tests

* feat(mojaloop/#3844): removed unneeded code

* feat(mojaloop/#3844): added testConsumer.clearEvents() for int-tests

* feat(mojaloop/#3844): skipped newly added int-test

* feat(mojaloop/#3844): updated validateFulfilCondition

* feat: unskip int-test

feat: unskip int-test

* feat(mojaloop/#3844): removed unneeded npm script

---------

Co-authored-by: Kevin Leyow <kleyow@gmail.com>

* test: added transferFulfilReject.end() (#1027)

* feat: fx fulfil position batching (#1019)

* feat: implemented fx

* fix: unit tests

* fix: unit tests

* chore: removed fx-fulfil in non batch mode

* feat: refactored position fulfil handler for fx

* chore: removed fx from non batch position fulfil

* chore: removed fx references from non batch position handler

* chore: simplified existing tests

* chore: added unit tests

* fix: prepare position fx

* chore(mojaloop/#3819): update functional tests and move fulfil int test (#1009)

* chore: update functional tests

* version

* snap

* test

* name

* func

* update

* test-function

* snap

* changes

* feat: implemented fx

* fix: unit tests

* fix: unit tests

* chore: removed fx-fulfil in non batch mode

* add back functions

* feat: refactored position fulfil handler for fx

* chore: removed fx from non batch position fulfil

* chore: removed fx references from non batch position handler

* chore: simplified existing tests

* chore: added unit tests

* fix: prepare position fx

* publish messages to batch topic

* update script

* move fxfulfil tests to batch tests

---------

Co-authored-by: Vijay <vijaya.guthi@infitx.com>

* chore: add integration tests for pos fulfil fx (#1030)

* feat: added integration test for fulfil fx

* chore: refined integration tests

* feat(mojaloop/#3818): added sequence and ER diagrams for transfer/fxTransfer flows (#1029)

* feat(mojaloop/#3818): added sequence and ER diagrams for transfer timeout cron

* feat(mojaloop/#3818): added sequence diagram for FX timeout

* feat(mojaloop/#3818): added sequence diagram for FX timeout

* feat(mojaloop/#3818): added transfer state diagrams

* feat(mojaloop/#3818): added transfer state diagrams

* feat(mojaloop/#3818): added transfer state diagrams

* feat(mojaloop/#3818): added transfer state diagrams

* feat(mojaloop/#3818): added transfer state diagrams

* feat(mojaloop/#3818): finalize FX timeout diagrams

* feat(mojaloop/#3818): added fxTimeout tables

* feat(mojaloop/#3903): update interal state on fx fulfil to RECEIVED_FULFIL_DEPENDENT (#1032)

* feat(mojaloop/#3903): update interal state on fx fulfil to RECEIVED_FULFIL_DEPENDENT

* file

* test

* chore: update harness (#1031)

* feat(mojaloop/#3904): add position event timeout reserved batch handling (#1033)

* chore: add integration test for batch

* unit

* reenable

* chore: comments

* cleanup function

* remove

* unskip

* fix potential int test failures

* reorder

* fix replace

* Update README.md

* feat: implemented timeout handler for fx (#1036)

* feat: added timeout handler implementation

* fix: queries

* fix: int tests

* fix: issues

* fix: fx timeout

* chore: update central services shared

* fix: cicd

* fix: deps

* fix: lint

* fix: unit tests

* chore: added unit tests

* chore: added unit tests

* Fix/test (#1039)

* dep, audit

* fix test

* chore: dep update

---------

Co-authored-by: Kevin Leyow <kleyow@gmail.com>

* feat: enable sending events directly to Kafka (#1037)

* chore(snapshot): 17.7.0-snapshot.3

* feat(mojaloop/#3904): add position event fx timeout reserved batch handling (#1035)

* chore: add integration test for batch

* unit

* reenable

* chore: comments

* feat: added timeout handler implementation

* test

* cleanup function

* test

* remove

* reorder

* fix potential int test failures

* fix tests

* fix: queries

* unit tests

* unskip

* fix potential int test failures

* reorder

* fix replace

* unit tests

* fix: int tests

* fix: issues

* fix: fx timeout

* chore: update central services shared

* fix: cicd

* fix: deps

* fix: lint

* fix: unit tests

* chore: added unit tests

* chore: added unit tests

* pull,audit,dep

* update tests

* update position query logic

* rename

* add comment

* detail

---------

Co-authored-by: Vijay <vijaya.guthi@infitx.com>

* chore(snapshot): 17.7.0-snapshot.4

* audit fix and dep update

* audit fix and dep update

* chore(snapshot): 17.7.0-snapshot.5

* image scan

* chore(snapshot): 17.7.0-snapshot.6

* image scan

* chore(snapshot): 17.7.0-snapshot.7

* node version

* chore(snapshot): 17.7.0-snapshot.8

* revert pipeline changes to get working snapshot

* chore(snapshot): 17.7.0-snapshot.9

* fix typo

* chore(snapshot): 17.7.0-snapshot.10

* fix command

* chore(snapshot): 17.7.0-snapshot.11

* fix: remove trx.rollback() (#1051)

* fix: produce followup messages in parallel (#1052)

* chore(snapshot): 17.7.0-snapshot.12

* fix: #3932 participant currency validation for fx (#1041)

* fix: validation

* fix: unit tests

* fix: int tests

* fix: some tests

* fix: tests

* fix: lint

* fix: integration tests

* chore: added coverage

* chore: lint

* chore: re-enabled some integration tests

* fix: avoid extra db call (#1055)

* chore(snapshot): 17.7.0-snapshot.13

* chore(snapshot): 17.7.0-snapshot.14

* feat(csi-164): parameterize switch id (#1057)

* fix: migration scripts

* chore(snapshot): 17.7.0-snapshot.17

* fix: migration scripts

* chore(snapshot): 17.7.0-snapshot.18

* chore(snapshot): 17.7.0-snapshot.19

* chore(snapshot): 17.7.0-snapshot.20

* feat: extend admin api to support proxy participants (#1043)

* feat: extend admin api to support proxy participants

* image revert

* audit

* changes

* changes

* add filter

* update swagger

* fix: allow isProxy

* feat(mojaloop/csi-190): add new state and functionality to handle proxied transfers (#1059)

* diff

* update diagram

* chore: happy path

* chore: int tests

* chore(snapshot): 17.8.0-snapshot.0

* update tests

* chore: add error cases

* test: coverage

* tests

* tests

* fix test

* update dep

---------

Co-authored-by: Kalin Krustev <kalin.krustev@gmail.com>

* feat(csi-22): add proxy lib to handlers (#1060)

* feat(csi-22): add proxy lib to handlers

* diff

* add

* int tests

* fix hanging int tests

* fixes?

* unit fixes?

* coverage

* feat: refactor proxy cache integration

* feat: restore default

* feat: minor optimization

* test: update coverage

* test: remove try-catch

* fix: fix disconnect error

* feat: proxy cache update (#1061)

* addressed comments

---------

Co-authored-by: Steven Oderayi <oderayi@gmail.com>

* feat(mojaloop/#3998): proxy obligation tracking for position changes (#1064)

* fix: consider HUB_ID when seeding the hub (#1073)

* fix: fsp id validation (#1074)

* fix: Cannot read properties of undefined

* chore(snapshot): 17.8.0-snapshot.1

* fix: name validation

* chore(snapshot): 17.8.0-snapshot.2

* fix: isProxy validation (#1075)

* feat(csi-22): add prepare participant substitution (#1065)

* unit tests

* unit tests

* int tests

* stuff

* some int tests

* comments

* pass object

* messy but working

* coverage

* hanging int test?

* fix int tests

* clarify naming

* comment

* fixes?

* dep update

* feat: fulfil obligation tracking (#1063)

* feat(csi-22): add proxy lib to handlers

* diff

* add

* int tests

* fix hanging int tests

* fixes?

* unit fixes?

* coverage

* feat: add zero adjustment for prepare position batch

* feat: refactor proxy cache integration

* feat: restore default

* feat: minor optimization

* test: update coverage

* test: remove try-catch

* fix: fix disconnect error

* feat(prepare-position): add proxy substitution and zero adjustment logic

* fix: remove uneeded async

* feat: proxy cache update (#1061)

* addressed comments

* chore: refactor

* test: add unit tests

* chore: minor refactor

* chore: lint

* feat: revert prepare hadnler change, update test coverage

* feat: update docker compose and default config for docker

* chore: remove commented code

* test: update test

* test: update test

* feat: added proxy check in fulfil handler

* fix: derive fn

* fix: checkSameCreditorDebtorProxy

* unit tests

* unit tests

* int tests

* fix: unit tests

* chore: added unit tests for proxyCache deriveCurrencyId function

* chore: added coverage

* stuff

* some int tests

* comments

* pass object

* messy but working

* coverage

* hanging int test?

* fix int tests

* feat: refactor

* clarify naming

* comment

* feat: added more test coverage

* fixes?

* dep update

* fix: int tests

* fix: disable tests around fspiop header validation in fulfil

* fix: int tests

* chore: disabled a fulfil test due to and issue in position handler

* fix: int tests

* chore: addressed pr comment

* fix: lint

* fix: integration tests

---------

Co-authored-by: Kevin Leyow <kleyow@gmail.com>
Co-authored-by: Steven Oderayi <oderayi@gmail.com>

* chore(snapshot): 17.8.0-snapshot.3

* feat(mojaloop/#3885): add migrations for storing fxQuotes (#1076)

* initial fxQuote migrations

* changes

* changes

* remove

* audit dep

* chore(snapshot): 17.8.0-snapshot.4

* feat: impl fx abort (#1077)

* feat: added abort batching

* fix: tests

* fix: bulk abort

* chore: cleanup

* chore: added int tests

* feat: added fx_abort_validation

* fix: positions

* fix: lint

* fix: unit tests

* chore: add test coverage

* fix: tests

* chore: migration fixes (#1078)

* chore: migration fixes

* chore(snapshot): 17.8.0-snapshot.5

* field

* chore(snapshot): 17.8.0-snapshot.6

* notnullable

* chore(snapshot): 17.8.0-snapshot.7

* feat: position zero messages (#1079)

* feat: initial commit

* fix: int tests

* fix: int tests

* chore: skip coverage check for snapshots

* chore(snapshot): 17.8.0-snapshot.8

* fix: proxy cluster

* chore: dep update

* chore(snapshot): 17.8.0-snapshot.9

* test: add unit tests for zero position change (#1081)

---------

Co-authored-by: Steven Oderayi <oderayi@gmail.com>

* chore: add logging for admin api, proxy lookup, participant domain (#1080)

* feat: initial commit

* fix: int tests

* fix: int tests

* chore: skip coverage check for snapshots

* chore(snapshot): 17.8.0-snapshot.8

* fix: proxy cluster

* chore: dep update

* chore(snapshot): 17.8.0-snapshot.9

* chore: add logging for admin api, proxy lookup, participant domain

* fix tests

* address comments

* ignore

---------

Co-authored-by: Vijay <vijaya.guthi@infitx.com>

* chore(snapshot): 17.8.0-snapshot.10

* fix: fx fulfil header validation (#1084)

* fix: fx fulfil

* chore(snapshot): 17.8.0-snapshot.10

* chore(snapshot): 17.8.0-snapshot.11

* chore(snapshot): 17.8.0-snapshot.12

* fix: fx fulfil proxy

* chore(snapshot): 17.8.0-snapshot.13

* ci: make redis cluster default for integration tests (#1083)

* fix: fx fulfil header validation2 (#1085)

* fix: added missing fields in query

* chore(snapshot): 17.8.0-snapshot.14

* fix: skip validation when payer and payee are represented by proxy (#1086)

* fix: test

* add unit tests

* update saveTransferPrepared

* chore: int tests

* retry count

---------

Co-authored-by: Vijay <vijaya.guthi@infitx.com>

* chore(snapshot): 17.8.0-snapshot.15

* feat(csi/551): add transfer state change for proxied fxTransfer (#1087)

* feat(csi/551): add transfer state change for proxied fxTransfer

* remove

* add case

* dep

* unit tests

* int tests

* chore(snapshot): 17.8.0-snapshot.16

* fix: remove misleading commit and rollback (#1089)

* fix: call commit and rollback

* test: fix coverage

* fix: call commit and rollback

* fix: remove unnecessary commit and rollback

* fix: remove unnecessary commit and rollback

* fix: duplicate fx transfers (#1097)

* fix: int tests

* fix: int tests

* fix: audit and lint fix

* fix: spelling

* chore: skipped an int test

* chore(snapshot): 17.8.0-snapshot.17

* chore(snapshot): 17.8.0-snapshot.18

* chore(snapshot): 17.8.0-snapshot.19

* chore(snapshot): 17.8.0-snapshot.20

* fix: add duplication logic and test for fxTransfers

* conversionState

* alter int test for message key 0

* alter test

---------

Co-authored-by: Kevin Leyow <kleyow@gmail.com>

* feat(csi-633): added externalParticipant model; added JSDocs; updated transfer/facade (#1099)

* feat(csi-318): added externalParticipants table

* refactor(csi-631): added calculateProxyObligation fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): improved logging in transfer facade

* chore(csi-632): added migrations to create externalParticipant table

* chore(csi-632): added migration to add externalParticipantId FK to fxTransferParticipant

* chore(csi-632): added migration to add externalParticipantId FK to fxTransferParticipant

* feat(csi-633): added externalParticipant model; added JSDocs; updated transfer/facade

* feat(csi-633): added externalParticipantId field to fxTransferParticipant table

* feat(csi-633): added externalParticipantId field to fxTransferParticipant table

* Revert "feat(csi-633): added externalParticipant model; added JSDocs; updated transfer/facade" (#1100)

Revert "feat(csi-633): added externalParticipant model; added JSDocs; updated…"

This reverts commit eb54f67.

* fix: get fx transfer not working (#1098)

* fix: int tests

* fix: int tests

* fix: audit and lint fix

* fix: spelling

* chore: skipped an int test

* chore(snapshot): 17.8.0-snapshot.17

* chore(snapshot): 17.8.0-snapshot.18

* chore(snapshot): 17.8.0-snapshot.19

* chore(snapshot): 17.8.0-snapshot.20

* fix: get fx transfers

* fix: refactor

* fix: fx fulfilment

* fix: unit tests

* fix: tests

* fix: fx transfer extension (#1102)

* fix: retify int tests (#1104)

* fix: fix abort callback (#1106)

fix: from argument in kafka notification for abort

* Revert "fix: fix abort callback" (#1109)

Revert "fix: fix abort callback (#1106)"

This reverts commit b6e9e2b.

* chore(snapshot): 17.8.0-snapshot.22

* fix: gp failure fixes for interscheme and fx changes (#1091)

* fix: check participant.isActive in prepare

* chore(snapshot): 17.8.0-snapshot.16

* chore(snapshot): 17.8.0-snapshot.17

* fix: check position account is active in prepare

* chore(snapshot): 17.8.0-snapshot.18

* test: temporarily disable coverage for proxy

* chore(snapshot): 17.8.0-snapshot.19

* chore(snapshot): 17.8.0-snapshot.20

* ci: temporarily disable int tests for snapshots

* chore(snapshot): 17.8.0-snapshot.21

* fix: fix typos

* refactor: reactor getFSPProxy

* chore(snapshot): 17.8.0-snapshot.22

* doc: update comment

* chore(snapshot): 17.8.0-snapshot.23

* fix(csi-603): fix getTransferParticipant query join

* ci: re-enable integration tests for snapshots

* chore(snapshot): 17.8.0-snapshot.24

* chore(snapshot): 17.8.0-snapshot.25

* fix: fix query

* chore(snapshot): 17.8.0-snapshot.26

* refactor: refactor

* refactor: refactor

* fix(csi-610): fix hub responding with RESERVED instead of COMMITED for v1.1 reserved fulfil

* chore(snapshot): 17.8.0-snapshot.27

---------

Co-authored-by: Vijay <vijaya.guthi@infitx.com>

* feat(csi/643): add fx-notify publishing on payer init fxTranfer success (#1105)

* feat(csi/643): add fx-notify publishing on payer init fxTranfer success

* loop

* deps

* tests

* list

* chore(snapshot): 17.8.0-snapshot.28

* fix: position changes (#1108)

* fix: from argument in kafka notification for abort

* fix: position changes

* fix: to number

* fix: position change in timeout

* fix: related fxtransfer check

* fix: unit tests

* fix: timeout

* chore: deps

* fix fx-abort tests

* fix fx-timeout tests

* chore: added a comment

* fix more tests

* fix: invalid fulfilment

* fix: unit test

* chore(snapshot): 17.8.0-snapshot.28

* chore(snapshot): 17.8.0-snapshot.29

* fix: lint

* chore(snapshot): 17.8.0-snapshot.30

---------

Co-authored-by: Kevin Leyow <kleyow@gmail.com>

* feat(csi-318): added externalParticipant table (#1092)

* feat(csi-318): added externalParticipants table

* refactor(csi-631): added calculateProxyObligation fn (#1093)

* refactor(csi-631): added calculateProxyObligation fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): improved logging in transfer facade

* chore(csi-632): added migrations to create externalParticipant table (#1094)

* refactor(csi-631): added calculateProxyObligation fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): improved logging in transfer facade

* chore(csi-632): added migrations to create externalParticipant table

* chore(csi-632): added migration to add externalParticipantId FK to fxTransferParticipant

* chore(csi-632): added migration to add externalParticipantId FK to fxTransferParticipant

* feat(csi-633): added externalParticipant model;  updated transfer/facade; added JSDocs; (#1101)

* refactor(csi-631): added calculateProxyObligation fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): added forwardPrepare fn

* refactor(csi-631): improved logging in transfer facade

* chore(csi-632): added migrations to create externalParticipant table

* chore(csi-632): added migration to add externalParticipantId FK to fxTransferParticipant

* chore(csi-632): added migration to add externalParticipantId FK to fxTransferParticipant

* feat(csi-633): added externalParticipant model; added JSDocs; updated transfer/facade

* feat(csi-633): added externalParticipantId field to fxTransferParticipant table

* feat(csi-633): added externalParticipantId field to fxTransferParticipant table

* feat(csi-633): updated from feat/fx-impl

* feat(csi-650): updated transferTimeout handler to take into account externalParticipant (#1107)

* feat(csi-650): updated transferTimeout handler to take into account externalParticipant

* feat(csi-650): fixed ep1.externalParticipantId field

* feat(csi-650): used leftJoin for externalParticipant table

* feat(csi-650): added externalPayeeName as source to timeout handler

* feat(csi-650): updated fxTimeout logic to take into account externalParticipant info

* feat(csi-650): code cleaning up

* feat(csi-650): code cleaning up

* feat(csi-651): updated fxAbort handling to use externalParticipant info (#1111)

* feat(csi-650): updated transferTimeout handler to take into account externalParticipant

* feat(csi-650): fixed ep1.externalParticipantId field

* feat(csi-650): used leftJoin for externalParticipant table

* feat(csi-650): added externalPayeeName as source to timeout handler

* feat(csi-650): updated fxTimeout logic to take into account externalParticipant info

* feat(csi-650): code cleaning up

* feat(csi-650): code cleaning up

* feat(csi-651): updated fxAbort handling to use externalParticipant info

* feat(csi-651): updated fxValidation handling

* feat(csi-651): fixed one leftJoin clause

* feat(csi-651): updated getExternalParticipantIdByNameOrCreate

* feat(csi-651): updated getExternalParticipantIdByNameOrCreate

* feat(csi-651): added externalParticipantCached model

* feat(csi-651): fixed prepare-internals tests

* feat(csi-651): added more tests

* feat(csi-651): reverted changes back to feat/fx-impl

* feat(csi-651): reverted unneeded changes back to feat/fx-impl version

* feat(csi-651): excluded some files from test coverage check

* chore(snapshot): 17.8.0-snapshot.32

* chore(snapshot): 17.8.0-snapshot.33

* feat(csi-634): added mock-knex lib to mock mysql in unit-tests (#1113)

* feat: add ULID support (#1114)

* Fix code scanning alert no. 9: Missing regular expression anchor

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix: uuid/ulid regex

* test: start using mojaloop/build orb (#1115)

* fix: sonar security hot spots

* Update src/shared/fspiopErrorFactory.js

Co-authored-by: shashi165 <33355509+shashi165@users.noreply.github.com>

* chore: dep audit

* chore: update licenses (#1138)

* chore: update licenses

* chore: update licenses

* chore: deps

* chore: address/remove todos

* chore: update state diagram

* chore: address comments

* chore: address comments

---------

Co-authored-by: Vijay <vijaya.guthi@infitx.com>
Co-authored-by: Eugen Klymniuk <keramet.kh@gmail.com>
Co-authored-by: vijayg10 <33152110+vijayg10@users.noreply.github.com>
Co-authored-by: Steven Oderayi <oderayi@gmail.com>
Co-authored-by: Kalin Krustev <kalin.krustev@gmail.com>
Co-authored-by: mojaloopci <info@mojaloop.io>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: shashi165 <33355509+shashi165@users.noreply.github.com>
  • Loading branch information
9 people authored Dec 20, 2024
1 parent 033ef63 commit aece4c7
Show file tree
Hide file tree
Showing 357 changed files with 28,877 additions and 9,701 deletions.
1,058 changes: 5 additions & 1,053 deletions .circleci/config.yml

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion .ncurc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,7 @@ reject: [
"get-port",
# sinon v17.0.1 causes 58 tests to fail. This will need to be resolved in a future story.
# Issue is tracked here: https://github.com/mojaloop/project/issues/3616
"sinon"
"sinon",
# glob >= 11 requires node >= 20
"glob"
]
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
18.17.1
18.20.3
17 changes: 16 additions & 1 deletion .nycrc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,20 @@ exclude: [
"**/node_modules/**",
'**/migrations/**',
'**/ddl/**',
'**/bulk*/**'
'**/bulk*/**',
'src/shared/logger/**',
'src/shared/loggingPlugin.js',
'src/shared/constants.js',
'src/domain/position/index.js',
'src/domain/position/binProcessor.js',
'src/handlers/positions/handler.js',
'src/handlers/transfers/createRemittanceEntity.js',
'src/handlers/transfers/FxFulfilService.js',
'src/models/position/batch.js',
'src/models/fxTransfer/**',
'src/models/participant/externalParticipantCached.js', # todo: figure out why it shows only 50% coverage in Branch
'src/models/transfer/facade.js', ## add more test coverage
'src/shared/fspiopErrorFactory.js',
'src/lib/proxyCache.js' # todo: remove this line after adding test coverage
]
## todo: increase test coverage before merging feat/fx-impl to main branch
19 changes: 9 additions & 10 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,27 @@ ARG NODE_VERSION=lts-alpine

# NOTE: Ensure you set NODE_VERSION Build Argument as follows...
#
# export NODE_VERSION="$(cat .nvmrc)-alpine" \
# docker build \
# --build-arg NODE_VERSION=$NODE_VERSION \
# -t mojaloop/central-ledger:local \
# . \
# export NODE_VERSION="$(cat .nvmrc)-alpine"
# docker build \
# --build-arg NODE_VERSION=$NODE_VERSION \
# -t mojaloop/central-ledger:local \
# .
#

# Build Image
FROM node:${NODE_VERSION} as builder
FROM node:${NODE_VERSION} AS builder

WORKDIR /opt/app

RUN apk --no-cache add git
RUN apk add --no-cache -t build-dependencies make gcc g++ python3 libtool openssl-dev autoconf automake bash \
RUN apk add --no-cache -t build-dependencies make gcc g++ python3 py3-setuptools libtool openssl-dev autoconf automake bash \
&& cd $(npm root -g)/npm \
&& npm install -g node-gyp

COPY package.json package-lock.json* /opt/app/

RUN npm ci
RUN npm prune --omit=dev

FROM node:${NODE_VERSION}
WORKDIR /opt/app
Expand All @@ -32,7 +33,7 @@ RUN mkdir ./logs && touch ./logs/combined.log
RUN ln -sf /dev/stdout ./logs/combined.log

# Create a non-root user: ml-user
RUN adduser -D ml-user
RUN adduser -D ml-user
USER ml-user

COPY --chown=ml-user --from=builder /opt/app .
Expand All @@ -43,7 +44,5 @@ COPY migrations /opt/app/migrations
COPY seeds /opt/app/seeds
COPY test /opt/app/test

RUN npm prune --production

EXPOSE 3001
CMD ["npm", "run", "start"]
48 changes: 39 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ Or via docker build directly:

```bash
docker build \
--build-arg NODE_VERSION="$(cat .nvmrc)-alpine" \
--build-arg NODE_VERSION="$(cat .nvmrc)-alpine3.19" \
-t mojaloop/ml-api-adapter:local \
.
```
Expand Down Expand Up @@ -113,21 +113,27 @@ NOTE: Only POSITION.PREPARE and POSITION.COMMIT is supported at this time, with

Batch processing can be enabled in the transfer execution flow. Follow the steps below to enable batch processing for a more efficient transfer execution:

Note: The position messages with action 'FX_PREPARE', 'FX_COMMIT' and 'FX_TIMEOUT_RESERVED' are only supported in batch processing.

- **Step 1:** **Create a New Kafka Topic**

Create a new Kafka topic named `topic-transfer-position-batch` to handle batch processing events.
- **Step 2:** **Configure Action Type Mapping**

Point the prepare handler to the newly created topic for the action type `prepare` using the `KAFKA.EVENT_TYPE_ACTION_TOPIC_MAP` configuration as shown below:
Point the prepare handler to the newly created topic for the action types those are supported in batch processing using the `KAFKA.EVENT_TYPE_ACTION_TOPIC_MAP` configuration as shown below:
```
"KAFKA": {
"EVENT_TYPE_ACTION_TOPIC_MAP" : {
"POSITION":{
"PREPARE": "topic-transfer-position-batch",
"BULK_PREPARE": "topic-transfer-position",
"COMMIT": "topic-transfer-position-batch",
"FX_COMMIT": "topic-transfer-position-batch",
"BULK_COMMIT": "topic-transfer-position",
"RESERVE": "topic-transfer-position",
"FX_PREPARE": "topic-transfer-position-batch",
"TIMEOUT_RESERVED": "topic-transfer-position-batch",
"FX_TIMEOUT_RESERVED": "topic-transfer-position-batch"
}
}
}
Expand Down Expand Up @@ -185,7 +191,8 @@ If you want to run integration tests in a repetitive manner, you can startup the
Start containers required for Integration Tests

```bash
docker-compose -f docker-compose.yml up -d mysql kafka init-kafka kafka-debug-console
source ./docker/env.sh
docker compose up -d mysql kafka init-kafka redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5
```

Run wait script which will report once all required containers are up and running
Expand Down Expand Up @@ -220,7 +227,8 @@ If you want to run integration tests in a repetitive manner, you can startup the
Start containers required for Integration Tests, including a `central-ledger` container which will be used as a proxy shell.

```bash
docker-compose -f docker-compose.yml -f docker-compose.integration.yml up -d kafka mysql central-ledger
source ./docker/env.sh
docker-compose -f docker-compose.yml -f docker-compose.integration.yml up -d kafka mysql central-ledger init-kafka redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5
```

Run the Integration Tests from the `central-ledger` container
Expand All @@ -235,24 +243,42 @@ If you want to run override position topic tests you can repeat the above and us

#### For running integration tests for batch processing interactively
- Run dependecies
```
docker-compose up -d mysql kafka init-kafka kafka-debug-console
```bash
source ./docker/env.sh
docker compose up -d mysql kafka init-kafka redis-node-0 redis-node-1 redis-node-2 redis-node-3 redis-node-4 redis-node-5
npm run wait-4-docker
```
- Run central-ledger services
```
nvm use
npm run migrate
env "CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__PREPARE=topic-transfer-position-batch" npm start
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__PREPARE=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__COMMIT=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__RESERVE=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__TIMEOUT_RESERVED=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__FX_TIMEOUT_RESERVED=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__ABORT=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__FX_ABORT=topic-transfer-position-batch
npm start
```
- Additionally, run position batch handler in a new terminal
```
nvm use
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__PREPARE=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__FX_PREPARE=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__COMMIT=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__TIMEOUT_RESERVED=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__FX_TIMEOUT_RESERVED=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__ABORT=topic-transfer-position-batch
export CLEDG_KAFKA__EVENT_TYPE_ACTION_TOPIC_MAP__POSITION__FX_ABORT=topic-transfer-position-batch
export CLEDG_HANDLERS__API__DISABLED=true
node src/handlers/index.js handler --positionbatch
```
- Run tests using `npx tape 'test/integration-override/**/handlerBatch.test.js'`
- Run tests using the following commands in a new terminal
```
nvm use
npm run test:int-override
```


If you want to just run all of the integration suite non-interactively then use npm run `test:integration`.
Expand All @@ -263,7 +289,11 @@ It will handle docker start up, migration, service starting and testing. Be sure
If you want to run functional tests locally utilizing the [ml-core-test-harness](https://github.com/mojaloop/ml-core-test-harness), you can run the following commands:

```bash
docker build -t mojaloop/central-ledger:local .
export NODE_VERSION="$(cat .nvmrc)-alpine"
docker build \
--build-arg NODE_VERSION=$NODE_VERSION \
-t mojaloop/central-ledger:local \
.
```

```bash
Expand Down
18 changes: 16 additions & 2 deletions audit-ci.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,20 @@
// Only use one of ["low": true, "moderate": true, "high": true, "critical": true]
"moderate": true,
"allowlist": [ // NOTE: Please add as much information as possible to any items added to the allowList
"GHSA-w5p7-h5w8-2hfq" // tap-spec>tap-out>trim; This has been analyzed and this is acceptable as it is used to run tests.
"GHSA-w5p7-h5w8-2hfq", // tap-spec>tap-out>trim; This has been analyzed and this is acceptable as it is used to run tests.
"GHSA-2mvq-xp48-4c77", // https://github.com/advisories/GHSA-2mvq-xp48-4c77
"GHSA-5854-jvxx-2cg9", // https://github.com/advisories/GHSA-5854-jvxx-2cg9
"GHSA-7hx8-2rxv-66xv", // https://github.com/advisories/GHSA-7hx8-2rxv-66xv
"GHSA-c429-5p7v-vgjp", // https://github.com/advisories/GHSA-c429-5p7v-vgjp
"GHSA-g64q-3vg8-8f93", // https://github.com/advisories/GHSA-g64q-3vg8-8f93
"GHSA-mg85-8mv5-ffjr", // https://github.com/advisories/GHSA-mg85-8mv5-ffjr
"GHSA-8hc4-vh64-cxmj", // https://github.com/advisories/GHSA-8hc4-vh64-cxmj
"GHSA-952p-6rrq-rcjv", // https://github.com/advisories/GHSA-952p-6rrq-rcjv
"GHSA-9wv6-86v2-598j", // https://github.com/advisories/GHSA-9wv6-86v2-598j
"GHSA-qwcr-r2fm-qrc7", // https://github.com/advisories/GHSA-qwcr-r2fm-qrc7
"GHSA-cm22-4g7w-348p", // https://github.com/advisories/GHSA-cm22-4g7w-348p
"GHSA-m6fv-jmcg-4jfg", // https://github.com/advisories/GHSA-m6fv-jmcg-4jfg
"GHSA-qw6h-vgh9-j6wx", // https://github.com/advisories/GHSA-qw6h-vgh9-j6wx
"GHSA-3xgq-45jj-v275" // High vulnerability https://github.com/advisories/GHSA-3xgq-45jj-v275 ignoring for now since devDependency
]
}
}
18 changes: 17 additions & 1 deletion config/default.json
Original file line number Diff line number Diff line change
Expand Up @@ -78,20 +78,36 @@
},
"INTERNAL_TRANSFER_VALIDITY_SECONDS": "432000",
"ENABLE_ON_US_TRANSFERS": false,
"PAYEE_PARTICIPANT_CURRENCY_VALIDATION_ENABLED": false,
"CACHE": {
"CACHE_ENABLED": false,
"MAX_BYTE_SIZE": 10000000,
"EXPIRES_IN_MS": 1000
},
"PROXY_CACHE": {
"enabled": true,
"type": "redis-cluster",
"proxyConfig": {
"cluster": [
{ "host": "localhost", "port": 6379 }
]
}
},
"API_DOC_ENDPOINTS_ENABLED": true,
"KAFKA": {
"EVENT_TYPE_ACTION_TOPIC_MAP" : {
"POSITION":{
"PREPARE": null,
"FX_PREPARE": "topic-transfer-position-batch",
"BULK_PREPARE": null,
"COMMIT": null,
"BULK_COMMIT": null,
"RESERVE": null
"RESERVE": null,
"FX_RESERVE": "topic-transfer-position-batch",
"TIMEOUT_RESERVED": null,
"FX_TIMEOUT_RESERVED": "topic-transfer-position-batch",
"ABORT": null,
"FX_ABORT": "topic-transfer-position-batch"
}
},
"TOPIC_TEMPLATES": {
Expand Down
92 changes: 90 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,22 @@
version: "3.7"

networks:
cl-mojaloop-net:
name: cl-mojaloop-net


# @see https://uninterrupted.tech/blog/hassle-free-redis-cluster-deployment-using-docker/
x-redis-node: &REDIS_NODE
image: docker.io/bitnami/redis-cluster:6.2.14
environment: &REDIS_ENVS
ALLOW_EMPTY_PASSWORD: yes
REDIS_CLUSTER_DYNAMIC_IPS: no
REDIS_CLUSTER_ANNOUNCE_IP: ${REDIS_CLUSTER_ANNOUNCE_IP}
REDIS_NODES: redis-node-0:6379 redis-node-1:9301 redis-node-2:9302 redis-node-3:9303 redis-node-4:9304 redis-node-5:9305
healthcheck:
test: [ "CMD", "redis-cli", "ping" ]
timeout: 2s
networks:
- cl-mojaloop-net

services:
central-ledger:
image: mojaloop/central-ledger:local
Expand Down Expand Up @@ -31,10 +44,14 @@ services:
- CLEDG_MONGODB__DISABLED=false
networks:
- cl-mojaloop-net
extra_hosts:
- "redis-node-0:host-gateway"
depends_on:
- mysql
- kafka
- objstore
- redis-node-0
# - redis
healthcheck:
test: ["CMD", "sh", "-c" ,"apk --no-cache add curl", "&&", "curl", "http://localhost:3001/health"]
timeout: 20s
Expand Down Expand Up @@ -94,6 +111,77 @@ services:
retries: 10
start_period: 40s
interval: 30s

redis-node-0:
<<: *REDIS_NODE
environment:
<<: *REDIS_ENVS
REDIS_CLUSTER_CREATOR: yes
REDIS_PORT_NUMBER: 6379
depends_on:
- redis-node-1
- redis-node-2
ports:
- "6379:6379"
- "16379:16379"
redis-node-1:
<<: *REDIS_NODE
environment:
<<: *REDIS_ENVS
REDIS_PORT_NUMBER: 9301
ports:
- "9301:9301"
- "19301:19301"
redis-node-2:
<<: *REDIS_NODE
environment:
<<: *REDIS_ENVS
REDIS_PORT_NUMBER: 9302
ports:
- "9302:9302"
- "19302:19302"
redis-node-3:
<<: *REDIS_NODE
environment:
<<: *REDIS_ENVS
REDIS_PORT_NUMBER: 9303
ports:
- "9303:9303"
- "19303:19303"
redis-node-4:
<<: *REDIS_NODE
environment:
<<: *REDIS_ENVS
REDIS_PORT_NUMBER: 9304
ports:
- "9304:9304"
- "19304:19304"
redis-node-5:
<<: *REDIS_NODE
environment:
<<: *REDIS_ENVS
REDIS_PORT_NUMBER: 9305
ports:
- "9305:9305"
- "19305:19305"

## To be used with proxyCache.type === 'redis'
# redis:
# image: redis:6.2.4-alpine
# restart: "unless-stopped"
# environment:
# <<: *REDIS_ENVS
# REDIS_CLUSTER_CREATOR: yes
# depends_on:
# - redis-node-1
# - redis-node-2
# - redis-node-3
# - redis-node-4
# - redis-node-5
# ports:
# - "6379:6379"
# networks:
# - cl-mojaloop-net

mockserver:
image: jamesdbloom/mockserver
Expand Down
9 changes: 9 additions & 0 deletions docker/central-ledger/default.json
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,15 @@
"MAX_BYTE_SIZE": 10000000,
"EXPIRES_IN_MS": 1000
},
"PROXY_CACHE": {
"enabled": true,
"type": "redis-cluster",
"proxyConfig": {
"cluster": [
{ "host": "redis-node-0", "port": 6379 }
]
}
},
"KAFKA": {
"TOPIC_TEMPLATES": {
"PARTICIPANT_TOPIC_TEMPLATE": {
Expand Down
Loading

0 comments on commit aece4c7

Please sign in to comment.