TTK Golden Path tests are failing on IaC central ledger v17.1.2

[sri-miriyala]

Summary:
When the Golden path is run against the latest version of IaC
Central Ledger (v17.1.2) , ML API adapter (v14.0.4), the TTK Golden Path is failing with the below error.
{
"errorInformation": {
"errorCode": "3100",
"errorDescription": "Generic validation error - fspiop-destination does not match payer fsp on the Fulfil callback response"
}
}

TTK Report artifacts (50).zip

Severity:
High

Priority:
Medium

Expected Behavior
Payer Callback FSP Destination equal to request FSP Source

Steps to Reproduce
The Central ledger and ML_API adapter versions used are below

1. moja-centralledger-service: mojaloop/central-ledger:v17.1.2
2. moja-ml-api-adapter-service: mojaloop/ml-api-adapter:v14.0.4
3. moja-ml-api-adapter-handler-notification: mojaloop/ml-api-adapter:v14.0.4
4. Run TTK golden path .
5. Check the results on P2P_MONEY_TRANSFER_PUT_NOTIFICATIONS - PAYEE RECEIVES NO NOTIFICATION WITH ABORTED STATUS AFTER SENDING INVALID FSPIOP-DESTINATION HEADER WITH TRANSFERSTATUS=COMMITTED
6. check the results on P2P_MONEY_TRANSFER_PATCH_NOTIFICATIONS - PAYEE RECEIVES PATCH NOTIFICATION WITH ABORTED STATUS AFTER SENDING INVALID FSPIOP-DESTINATION HEADER WITH TRANSFERSTATUS=RESERVED

Specifications

• Component (if known): CL, ML API adapter
• Version:
• Platform:
• Subsystem:
• Type of testing: Functional
• Bug found/raised by: @sri-miriyala , Oscar

Notes:
The following are the versions of components in IaC

bof-reporting-events-processor-svc: docker.io/mojaloop/reporting-events-processor-svc:v2.0.0,
bof-reporting-hub-bop-api-svc: docker.io/mojaloop/reporting-hub-bop-api-svc:v2.0.3,
bof-reporting-hub-bop-experience-api-svc: docker.io/mojaloop/reporting-hub-bop-experience-api-svc:v2.0.2,mojaloop/event-sidecar:v11.0.1,
bof-reporting-hub-bop-positions-ui: docker.io/mojaloop/reporting-hub-bop-positions-ui:v0.0.6,
bof-reporting-hub-bop-role-ui: docker.io/mojaloop/reporting-hub-bop-role-ui:v1.5.3,
bof-reporting-hub-bop-settlements-ui: docker.io/mojaloop/reporting-hub-bop-settlements-ui:v0.0.17,
bof-reporting-hub-bop-shell: docker.io/mojaloop/reporting-hub-bop-shell:v2.1.1,
bof-reporting-hub-bop-trx-ui: docker.io/mojaloop/reporting-hub-bop-trx-ui:v1.7.2,
bof-reporting-legacy-api: docker.io/mojaloop/reporting:v11.0.0,
bof-role-assignment-service: docker.io/mojaloop/role-assignment-service:v1.0.1,
bof-security-hub-bop-kratos-ui: ghcr.io/modusbox/oidcer:latest,
bof-security-role-perm-operator-svc: docker.io/mojaloop/security-role-perm-operator-svc:v2.0.0,
bulk-mongodb: docker.io/bitnami/mongodb:6.0.2-debian-11-r11,
cep-mongodb: docker.io/bitnami/mongodb:6.0.2-debian-11-r11,
keto: oryd/keto:v0.6.0-alpha.1-sqlite,
kratos: oryd/kratos:v0.5.5-alpha.1,
mfi-account-oracle: ghcr.io/mojaloop/mfi-account-oracle:2.0.4,
mfi-p2p-oracle: ghcr.io/mojaloop/mfi-account-oracle:2.0.4,
moja-account-lookup-service-admin: mojaloop/account-lookup-service:v14.1.0,mojaloop/event-sidecar:v12.0.0,
moja-account-lookup-service: mojaloop/account-lookup-service:v14.2.3,mojaloop/event-sidecar:v12.0.0,
moja-auth-svc: mojaloop/auth-service:v14.0.0,
moja-bulk-api-adapter-handler-notification: mojaloop/bulk-api-adapter:v15.0.3,
moja-bulk-api-adapter-service: mojaloop/bulk-api-adapter:v15.0.3,
moja-centralledger-handler-admin-transfer: mojaloop/central-ledger:v17.1.2,
moja-centralledger-handler-timeout: mojaloop/central-ledger:v17.1.2,mojaloop/event-sidecar:v12.0.0,
moja-centralledger-handler-transfer-fulfil: mojaloop/central-ledger:v17.1.2,mojaloop/event-sidecar:v12.0.0,
moja-centralledger-handler-transfer-get: mojaloop/central-ledger:v17.1.2,
moja-centralledger-handler-transfer-position: mojaloop/central-ledger:v17.1.2,mojaloop/event-sidecar:v12.0.0,
moja-centralledger-handler-transfer-prepare: mojaloop/central-ledger:v17.1.2,mojaloop/event-sidecar:v12.0.0,
moja-centralledger-service: mojaloop/central-ledger:v17.1.2,
moja-centralsettlement-handler-deferredsettlement: mojaloop/central-settlement:v15.0.0,mojaloop/event-sidecar:v12.0.0,
moja-centralsettlement-handler-grosssettlement: mojaloop/central-settlement:v15.0.0,mojaloop/event-sidecar:v12.0.0,
moja-centralsettlement-handler-rules: mojaloop/central-settlement:v15.0.0,mojaloop/event-sidecar:v12.0.0,
moja-centralsettlement-service: mojaloop/central-settlement:v15.0.0,mojaloop/event-sidecar:v12.0.0,
moja-cl-handler-bulk-transfer-fulfil: mojaloop/central-ledger:v17.0.3,
moja-cl-handler-bulk-transfer-get: mojaloop/central-ledger:v17.0.3,
moja-cl-handler-bulk-transfer-prepare: mojaloop/central-ledger:v17.0.3,
moja-cl-handler-bulk-transfer-processing: mojaloop/central-ledger:v17.0.3,
moja-consent-oracle: mojaloop/als-consent-oracle:v0.2.0,
moja-ml-api-adapter-handler-notification: mojaloop/ml-api-adapter:v14.0.4,mojaloop/event-sidecar:v12.0.0,
moja-ml-api-adapter-service: mojaloop/ml-api-adapter:v14.0.4,mojaloop/event-sidecar:v12.0.0,
moja-ml-testing-toolkit-frontend: mojaloop/ml-testing-toolkit-ui:v15.4.0,
moja-quoting-service: mojaloop/quoting-service:v15.0.2,mojaloop/event-sidecar:v12.0.0,
moja-sim-payeefsp-backend: mojaloop/mojaloop-simulator:v14.0.1,
moja-sim-payeefsp-cache: redis:5.0.4-alpine,
moja-sim-payeefsp-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-payerfsp-backend: mojaloop/mojaloop-simulator:v14.0.1,
moja-sim-payerfsp-cache: redis:5.0.4-alpine,
moja-sim-payerfsp-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-testfsp1-backend: mojaloop/mojaloop-simulator:v14.0.1,
moja-sim-testfsp1-cache: redis:5.0.4-alpine,
moja-sim-testfsp1-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-testfsp2-backend: mojaloop/mojaloop-simulator:v14.0.1,
moja-sim-testfsp2-cache: redis:5.0.4-alpine,
moja-sim-testfsp2-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-testfsp3-backend: mojaloop/mojaloop-simulator:v14.0.1,
moja-sim-testfsp3-cache: redis:5.0.4-alpine,
moja-sim-testfsp3-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-testfsp4-backend: mojaloop/mojaloop-simulator:v14.0.1,
moja-sim-testfsp4-cache: redis:5.0.4-alpine,
moja-sim-testfsp4-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-tp-dfspa-backend: mojaloop/mojaloop-simulator:v13.0.0,
moja-sim-tp-dfspa-cache: redis:5.0.4-alpine,
moja-sim-tp-dfspa-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-tp-dfspa-thirdparty-sdk: mojaloop/thirdparty-sdk:v15.1.0,
moja-sim-tp-dfspb-backend: mojaloop/mojaloop-simulator:v13.0.0,
moja-sim-tp-dfspb-cache: redis:5.0.4-alpine,
moja-sim-tp-dfspb-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-tp-dfspb-thirdparty-sdk: mojaloop/thirdparty-sdk:v15.1.0,
moja-sim-tp-pisp-backend: mojaloop/mojaloop-simulator:v13.0.0,
moja-sim-tp-pisp-cache: redis:5.0.4-alpine,
moja-sim-tp-pisp-scheme-adapter: mojaloop/sdk-scheme-adapter:v23.0.1,
moja-sim-tp-pisp-thirdparty-sdk: mojaloop/thirdparty-sdk:v15.1.0,
moja-simulator: mojaloop/simulator:v12.0.0,
moja-tp-api-svc: mojaloop/thirdparty-api-svc:v13.0.2,
moja-transaction-requests-service: mojaloop/transaction-requests-service:v14.1.0,mojaloop/event-sidecar:v12.0.0,
moja-ttksim1-sdk-api-svc: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim1-sdk-cmd-evt-handler: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim1-sdk-dom-evt-handler: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim1-ttk-frontend: mojaloop/ml-testing-toolkit-ui:v15.4.0,
moja-ttksim2-sdk-api-svc: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim2-sdk-cmd-evt-handler: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim2-sdk-dom-evt-handler: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim2-ttk-frontend: mojaloop/ml-testing-toolkit-ui:v15.4.0,
moja-ttksim3-sdk-api-svc: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim3-sdk-cmd-evt-handler: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim3-sdk-dom-evt-handler: docker.io/mojaloop/sdk-scheme-adapter:v23.0.1,
moja-ttksim3-ttk-frontend: mojaloop/ml-testing-toolkit-ui:v15.4.0,
oathkeeper-maester: oryd/oathkeeper-maester:v0.1.5,
oathkeeper: oryd/oathkeeper:v0.38.9-beta.1,
publicapi-oathkeeper: oryd/oathkeeper:v0.38.9-beta.1,
reporting-events-mongodb: docker.io/bitnami/mongodb:6.0.2-debian-11-r11,
ttk-mongodb: docker.io/bitnami/mongodb:6.0.2-debian-11-r11,

[PaulGregoryBaker]

Hey team! Please add your planning poker estimate with Zenhub @aaronreynoza @kleyow @oderayi @sri-miriyala @vijayg10

[aaronreynoza]

Issues described on this bug were fixed in #3580. However running the GP on the IaC environment shows new errors related to removing websockets from the TTK. Wait time is reduced, therefore some tests fail randomly when a transfer is made and the callback is made to the Simulator.

[sri-miriyala]

This is the test results after the fix from Aaron.