Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm charts incompatible with Kubernetes v1.16.x #219

Closed
mdebarros opened this issue Nov 4, 2019 · 11 comments
Closed

Helm charts incompatible with Kubernetes v1.16.x #219

mdebarros opened this issue Nov 4, 2019 · 11 comments

Comments

@mdebarros
Copy link
Member

mdebarros commented Nov 4, 2019

Summary:
Helm charts may be incompatible with Kubernetes v1.16.x due to deprecated APIs as per the following link: https://kubernetes.io/blog/2019/07/18/api-deprecations-in-1-16/

The following resource kinds will need to be migrated:

  • NetworkPolicy (in the extensions/v1beta1 API group)
    Migrate to use the networking.k8s.io/v1 API, available since v1.8. Existing persisted data can be retrieved/updated via the networking.k8s.io/v1 API.
  • PodSecurityPolicy (in the extensions/v1beta1 API group)
    Migrate to use the policy/v1beta1 API, available since v1.10. Existing persisted data can be retrieved/updated via the policy/v1beta1 API.
  • DaemonSet, Deployment, StatefulSet, and ReplicaSet (in the extensions/v1beta1 and apps/v1beta2 API groups)
    Migrate to use the apps/v1 API, available since v1.9. Existing persisted data can be retrieved/updated via the apps/v1 API.

The following resource kinds need to be maintained:

  • Ingress (in the extensions/v1beta1 API group) SHOULD NOT be upgraded at this time since it will impact backward compatibility < 1.14.x. This will need to be migrated at a later point once v1.20.x reaches critical mass.

Severity:
High

Priority:
Critical

Expected Behavior

Deployments should be successful on v1.16x, v1.15x, v1.14x and v1.11x

Steps to Reproduce

  1. Deploy Mojaloop Helm chart to v1.16.x Kubernetes cluster

Specifications

  • Component (if known): Mojaloop Helm Charts
  • Version: v8.x.x
  • Platform: Helm
  • Subsystem: n/a
  • Type of testing: Manual
  • Bug found/raised by: @vijayg10

Notes:

  • Severity when opened: High
  • Priority when opened: Critical
  • Fixes should be tested against Kubernetes v1.11.x, v1.14 & v1.15.x to ensure backward compatibility,
  • Alternatively, there will need to be a config param that sets the apiVersion appropriately for backward compatibility to be maintained, assuming the update apiVersion in v1.16.x is not supported in v.15.x
@mdebarros
Copy link
Member Author

I think we need to possibly include versions 1.17-1.18 for testing. Or at least pick up an issues that need to be further resolved going forward.

@lewisdaly
Copy link
Contributor

I think @tdaly61 might be taking this one up. I'll assign him once he's accepted the github invite.

@tdaly61
Copy link
Contributor

tdaly61 commented Apr 3, 2020

happy to take it ..please assign

@lewisdaly
Copy link
Contributor

Looks like we might have issues with percona-xtradb-cluster. From what I've seen so far, the latest version 1.0.3 was last updated a year ago, and requires the old beta apis.

https://github.com/helm/charts/tree/master/stable/percona-xtradb-cluster

@lewisdaly
Copy link
Contributor

Specifically, it requires the StatefulSet resource at apiVersion: apps/v1beta2:

https://github.com/helm/charts/blob/master/stable/percona-xtradb-cluster/templates/statefulset.yaml

@lewisdaly
Copy link
Contributor

lewisdaly commented Apr 16, 2020

Relevant issues on the helm charts repo:

helm/charts#19697
helm/charts#21538

I want to see if we can do a PR into the helm/charts repo, but will will likely take a while to get across the line.

@lewisdaly
Copy link
Contributor

The following (very rough) Draft PR (based on the helm3 changes) is working for kubernetes v1.17: https://github.com/vessels-tech/helm/tree/fix/219-kubernetes-17

@tdaly61
Copy link
Contributor

tdaly61 commented Apr 16, 2020

The following (very rough) Draft PR (based on the helm3 changes) is working for kubernetes v1.17: https://github.com/vessels-tech/helm/tree/fix/219-kubernetes-17

Also I have it running on microk8s v.1.8 , I have data loading without error but there "appear" to be some issues with the Golden_Path tests. I am not familiar enough yet with the application to know how to debug this, and I think @lewisdaly will look at it in the near term with me.

@lewisdaly
Copy link
Contributor

It looks like there is a PR in progress for the percona charts: helm/charts#22236

@mdebarros
Copy link
Member Author

mdebarros commented Jun 15, 2020

helm/charts#22236

It was merged 10 days ago! :D/

I think we should do a similar change to our charts using the Capabilities.APIVersions.Has, and add a helper function to determine the version.

@mdebarros mdebarros changed the title Helm charts may be incompatible with Kubernetes v1.16.x Helm charts incompatible with Kubernetes v1.16.x Sep 10, 2020
@mdebarros
Copy link
Member Author

The issue has been solved in master branch with the following PR: #349

It will be included as part of the release v11.x of the Mojaloop Helm Charts.

mdebarros added a commit that referenced this issue Nov 25, 2020
## Helm release changes

### 1. Maintenance updates
1. GitHub issue: mojaloop/project#1617
2. FSP Interoperability API v1.1 Swagger file used from here: https://github.com/mojaloop/mojaloop-specification/tree/master/documents/v1.1-document-set
3. For breaking changes, please review the "Breaking Changes" section below for for Mojaloop Helm v11.0.0
4. Notes on simulators 
	- [Mojaloop-Simulator](https://github.com/mojaloop/mojaloop-simulator) is enabled by default (six instances used). 
	- Ensure that correct Postman Scripts are used if you wish to test against the Mojaloop-Simulators:
        	-  A single MojaSim collection is now available after reconciliation between two collections that existed previously.
        	- Setup Mojaloop Hub : [MojaloopHub_Setup](https://github.com/mojaloop/postman/blob/master/MojaloopHub_Setup.postman_collection.json)
        	- Setup Mojaloop Simulators for testing : [MojaloopSims_Onboarding](https://github.com/mojaloop/postman/blob/master/MojaloopSims_Onboarding.postman_collection.json)
        	- Golden path tests: [Golden_Path_Mojaloop](https://github.com/mojaloop/postman/blob/master/Golden_Path_Mojaloop.postman_collection.json)
	- Legacy Simulators are still required and deployed by default; disabling this will cause issues since there is Account Lookup directory mocking functionality in this service. To run Golden path tests using the Legacy simulator:
                - Setup Mojaloop Hub, test FSPs : [ML_OSS_Setup_LegacySim](https://github.com/mojaloop/postman/blob/master/ML_OSS_Setup_LegacySim.postman_collection.json)
        	- Golden path tests: [ML_OSS_Golden_Path_LegacySim](https://github.com/mojaloop/postman/blob/master/ML_OSS_Golden_Path_LegacySim.postman_collection.json)
	- In order to run the Legacy Simulator postman collection or the Mojaloop-Simulator collection, their corresponding setup collection needs to be run first; this can be done without redeployment.
5. Upgrades to all core and supporting services to support 11.0.0 release _(see Application versions and release notes below)_
6. The current Golden Path collection expects
 `quoting-service.config.simple_routing_mode_enabled` to be set to `false` to get 100% pass-rate, if this flag is disabled, there will be failures in several quoting-service tests (7 expected failures). This command can be used to override the default config: `helm install moja ./mojaloop/ -n demo --set quoting-service.config.simple_routing_mode_enabled=false`
7. Along with this configuration for on-us transfers is disabled on both default deployments and postman (ON_US_TRANSFERS_ENABLED on postman) collections.
8. Moving to SemVer as the standard is now adopted. This will start with v11.0.0 and follow the general semantic versioning guidance after that.
9. Updated Promfana chart with "Prometheus: v2.20.1; Grafana: v7.1.1"
10. Updated EFK chart with “appVersion: "elasticsearch: 7.7.1; kibana: 7.7.1; apm-server: 7.7.1; fluentd-elasticsearch: 3.0.4"
11. Charts now support deployments on Kubernetes v1.16.x onwards - Fix for issue #219
12. Removed deprecated ingress-nginx chart (#350). Please use [Nginx official chart](https://docs.nginx.com/nginx-ingress-controller/installation/installation-with-helm/).
13. Updated official Helm repo URIs for stable & incubator dependencies and documentation: 
	- Stable: https://kubernetes-charts.storage.googleapis.com/ --> https://charts.helm.sh/stable
	- Incubator: http://storage.googleapis.com/kubernetes-charts-incubator --> https://charts.helm.sh/incubator
13. Updated `innodb_flush_log_at_trx_commit` to the value of `1` for MySQL servers to ensure ACID compliance (ref: https://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit)
14. Renamed Central-Event-Processor's (CEP) MongoDB instance dependency to: `nameOverride: cep-mongodb`. This will ensure that the CEP has its own MongoDB instance.
15. Renamed Mojaloop-Self-Testing-Toolkit's (TTK) MongoDB instance dependency to: `nameOverride: ttk-mongodb`. This will ensure that the TTK has its own MongoDB instance.


### 2. New Features
1. Along with changes from v10.4.0 related to the version v1.1 of the Mojaloop FSP Interoperability API, additional follow-up changes are implemented and a few fixes made. The changes are listed in the [tracking issue](mojaloop/mojaloop-specification#52) of the specification and there is additional context offered [here](https://mojaloop.io/mojaloop-fsp-interoperability-api-version-1-1/). Epic: mojaloop/project#1333
2. Added TTK CLI chart to ml-testing-toolkit (#380).
3. Added the option to pass a template as the rules.json (#353) in quoting service
4. Updates for bulk transfers support in MojaSims/Scheme-Adapter (mojaloop/project/issues/1386)
5. Added charts for bulk-get-handler: mojaloop/project/issues/1496
6. Package development versions (#354)
7. Update helm charts to support kubernetes deployments1.15 and later versions: mojaloop/project/issues/1463 (PR#349)
8. Added TTK CLI chart to ml-testing-toolkit (#380)
9. Modified the testing toolkit CLI chart to enabled scheduling (#381) 


### 3. Bug Fixes
1. Incorrect error behaviour on transfer fulfilment with future timestamp [1708](mojaloop/project#1708)
2. ACCEPT header returned in the PUT from switch for the timeout scenario [1710](mojaloop/project#1710) 
3. Date header missing from the Switch signature in timeout scenario [1709](mojaloop/project#1709)  
4. Valid transactionRequests callback in SDK responding with error response [1647](mojaloop/project#1647)  
5. Support test currencies XXX, XTS in ml-api-adapter [1652](mojaloop/project#1652)

## 4. Application versions
Application versions that are supported for this update:
1. ml-api-adapter: v10.5.0 -> **v11.1.2**
2. central-ledger: v10.5.1 -> **v11.3.1**
3. account-lookup-service: v10.4.2 -> **v11.1.2**
4. quoting-service: v10.5.5 -> **v11.1.4**
5. central-settlement: **v10.5.0** (no change)
6. central-event-processor: **v10.5.0** (no change)
7. bulk-api-adapter: v10.5.0 -> **v11.0.2**
8. email-notifier: **v9.5.0** (no change)
9. als-oracle-pathfinder: v9.4.0 --> **v10.2.0**
10. transaction-requests-service: v10.4.0 -> **v11.1.2**
11. finance-portal-ui: **v10.4.0** (no change)
12. finance-portal-backend-service: **v10.4.0** (no change)
13. settlement-management: **v8.8.2** (no change)
14. operator-settlement: **v9.2.1**  (no change)
15. event-sidecar: **v9.5.1**  (no change)
16. event-stream-processor: **v9.5.0-snapshot**
17. simulator: v10.5.3 -> **v11.0.2**
18. mojaloop-simulator: v10.4.1 -> **v11.2.1**
19. sdk-scheme-adapter: v10.5.0 -> **v11.8.0**
20. ml-testing-toolkit: v10.4.1 -> **v11.5.0**
21. ml-testing-toolkit-ui: v10.4.1 -> **v11.5.0**
22. ml-testing-toolkit-cli: **v11.3.0** (new)
 
## 5. Application release notes
1. ml-api-adapter - https://github.com/mojaloop/ml-api-adapter/releases/tag/v11.1.2
2. central-ledger - https://github.com/mojaloop/central-ledger/releases/tag/v11.3.1
3. account-lookup-service - https://github.com/mojaloop/account-lookup-service/releases/tag/v11.1.2
4. quoting-service - https://github.com/mojaloop/quoting-service/releases/tag/v11.1.4
5. central-settlement- https://github.com/mojaloop/central-settlement/releases/tag/v10.5.0
6. central-event-processor - https://github.com/mojaloop/central-event-processor/releases/tag/v10.5.0
7. bulk-api-adapter - https://github.com/mojaloop/bulk-api-adapter/releases/tag/v11.0.2
8. email-notifier - https://github.com/mojaloop/email-notifier/releases/tag/v9.5.0
9. als-oracle-pathfinder - https://github.com/mojaloop/als-oracle-pathfinder/releases/tag/v10.2.0
10. transaction-requests-service - https://github.com/mojaloop/transaction-requests-service/releases/tag/v11.1.2
11. finance-portal-ui - https://github.com/mojaloop/finance-portal-ui/releases/tag/v10.4.0
12. finance-portal-backend-service - https://github.com/mojaloop/finance-portal-backend-service/releases/tag/v10.4.0
13. settlement-management - https://github.com/mojaloop/settlement-management/releases/tag/v8.8.2
14. operator-settlement - https://github.com/mojaloop/operator-settlement/releases/tag/v9.2.1
15. event-sidecar - https://github.com/mojaloop/event-sidecar/releases/tag/v9.5.1
16. event-stream-processor - https://github.com/mojaloop/event-stream-processor/releases/v9.5.0-snapshot
17. simulator - https://github.com/mojaloop/simulator/releases/tag/v11.0.2
18. mojaloop-simulator - https://github.com/mojaloop/mojaloop-simulator/releases/tag/v11.2.1
19. sdk-scheme-adapter - https://github.com/mojaloop/sdk-scheme-adapter/releases/tag/v11.8.0
20. ml-testing-toolkit - https://github.com/mojaloop/ml-testing-toolkit/releases/tag/v11.5.0
21. ml-testing-toolkit-ui - https://github.com/mojaloop/ml-testing-toolkit-ui/releases/tag/v11.5.0
22. ml-testing-toolkit-cli - https://github.com/mojaloop/ml-testing-toolkit-ui/releases/tag/v11.3.0
 
## 6. Operational Chart versions
- EFK: **v7.1.0**  -> **v11.0.0**
- Promfana: **v8.4.0** -> **v11.0.0**

## 7. Breaking changes
- Helm release v10.4.0 is not upgradable to v11.0.0 - mojaloop/project#1759

## 8. Known Issues
- Proposal to remove external dependencies from Mojaloop Helm Charts - mojaloop/design-authority-project#69 **(_Note: this will be a breaking change in the next major release_)**
- ~Testing Toolkit Post-Hook Install tests are not compatible with Postman (_Note: this will not impact default installations_) - mojaloop/project#1864


## 9. Contributors:

- Contributing organizations: BMGF, CrossLake, MBX 
- Crosslake: @lewisdaly
- ModusBox: @amarmodus, @elnyry-sam-k , @mdebarros, @oderayi, @rmothilal , @vgenev , @vijayg10, @msk-, @vbarzokas, @aaronreynoza, @KamuelaFranco
- Individuals:  @vorburger
_Note: companies in alphabetical order_

---
## PR Dependencies
- mojaloop/project#1866
- mojaloop/postman#188
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants