-
Notifications
You must be signed in to change notification settings - Fork 253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Race condition between reading the ack-role-account-map ConfigMap and resyncing breaks CARM #2011
Comments
@nampnguyen Is this happening only during controller startup, or have you observed it persisting even after multiple reconciliations? We can change a few things in the ACK runtime to mitigate this:
Quick note: We really need to start stress and load testing for ACK controllers, especially for components like CARM watchers.. i'm pretty sure there is a lot we can catch in there. |
@a-hilaly In our experience, if the reconciler starts before the When we originally saw this issue when we had a CPU limit of |
…tated namespaces (#138) Addresses aws-controllers-k8s/community#2011 In certain scenarios, where a user deploys a resource to a namespace annotated with a specific ownner accountID, a race condition was identified between the reconciler and the CARM (Cross Account Resource Management) `ConfigMap`. This race condition resulted in the controller setting an empty roleARN, preventing the aws-sdk-go client from pivoting (calling `STS::AssumeRole`) and managing resourecs in the correct account. Instead, resources were inadvertently managed in the default account instead of the namespace assigned account. This issue stemmed from the initial implementation of the CARM feature, where the method responsible for retrieving the accountID from the cache, didn't not properly verify the existance and content of the CARM configMap and instead returned an empty stringy when these conditions were not satisfied. This led to selection of the default account (when an empty `RoleARN` is returned )for resource management. Although these scenarios are rare, they can occur in clusters with a significantly high number of namespaces, causing a delay between naemsapce/configmap events and the informer's event handlers. This patch addresses the race issue by implementing two main things: - Proper error propagation: an error is no propagated when a `ConfigMap` is missing or when an accountID entry is missing in the `ConfigMap`. This helps the reconciler make the right decision on how to handle these cases. - Improved error handling: The reconciler now carefully handles these errors and requeues whenever a user has issued an owneraccountid-annotated namespace but the Configmap is not create or properly propagated. Signed-off-by: Amine Hilaly <hilalyamine@gmail.com> By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@nampnguyen we starting rolling out a fix for all the controllers - i'll ping here once dynamodb controller is patched. |
Pin ACK runtime to v0.31.0. Mainly addressing two github issues: - Healthz, liveness and readiness probes aws-controllers-k8s/community#2012 - CARM Race condition and scaling issues aws-controllers-k8s/community#2011 By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@a-hilaly Saw the controllers PRs that included the |
@nampnguyen Looks like we also needed aws-controllers-k8s/runtime@861f7ed - we're rolling a new patch today :) |
@a-hilaly We tried the latest controllers using runtime
Even waiting several minutes, I'm not seeing the re-queuing behavior described in the PRs. |
@nampnguyen I will try to reproduce locally, maybe i'm missing something in here... |
@a-hilaly Yes, the namespaces do have the If I'm reading this correctly, this line here assumes that if the resource's Account ID is the same as the controller's IRSA account ID, then CARM is not used. In the most recent logs I shared, the resource's Account ID and Controller account are in fact the same, but we still need the CARM role pivot. I think this can be fixed by always checking for the namespace annotation and if present use CARM. |
@nampnguyen makes sense~ working on a fix! I think we might also want to enable CARM even in single-namespace watch mode.
Correct. Not sure, but this might have been the case even before. |
…tated namespaces (aws-controllers-k8s#138) Addresses aws-controllers-k8s/community#2011 In certain scenarios, where a user deploys a resource to a namespace annotated with a specific ownner accountID, a race condition was identified between the reconciler and the CARM (Cross Account Resource Management) `ConfigMap`. This race condition resulted in the controller setting an empty roleARN, preventing the aws-sdk-go client from pivoting (calling `STS::AssumeRole`) and managing resourecs in the correct account. Instead, resources were inadvertently managed in the default account instead of the namespace assigned account. This issue stemmed from the initial implementation of the CARM feature, where the method responsible for retrieving the accountID from the cache, didn't not properly verify the existance and content of the CARM configMap and instead returned an empty stringy when these conditions were not satisfied. This led to selection of the default account (when an empty `RoleARN` is returned )for resource management. Although these scenarios are rare, they can occur in clusters with a significantly high number of namespaces, causing a delay between naemsapce/configmap events and the informer's event handlers. This patch addresses the race issue by implementing two main things: - Proper error propagation: an error is no propagated when a `ConfigMap` is missing or when an accountID entry is missing in the `ConfigMap`. This helps the reconciler make the right decision on how to handle these cases. - Improved error handling: The reconciler now carefully handles these errors and requeues whenever a user has issued an owneraccountid-annotated namespace but the Configmap is not create or properly propagated. Signed-off-by: Amine Hilaly <hilalyamine@gmail.com> By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
@a-hilaly @nampnguyen We are also facing this exact issue when using most recent version of iam-controller. Glad to hear a fix is being worked on. Thanks! |
@a-hilaly we do still face this same issue even after your pull request from April. Just wondering if someone is looking at this problem still? |
@mattzech @mumlawski @nampnguyen Sorry folks I somehow got side tracked from this issue. This is a high priority issue and I i'll ship a fix for it ASAP. |
@mumlawski @mattzech @nampnguyen we released a new iam-controller version fix the CARM/CM race condition issue |
/close |
@a-hilaly: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Describe the bug
When a controller starts while using CARM, particularly if the pod has a CPU limit specified (~250m), the account config map can sometimes be read after reconciliation starts. When this happens, the controller no longer assumes the appropriate cross account role and results in AccessDenied errors because the IRSA role does not have permissions to resources in other accounts.
The problem seems to be getting worse over time and may be related to the number of namespaces.
Even after the
ack-role-account-map
is read, the controller does not use the correct cross account role.Debug logs (with some redactions):
Steps to reproduce
Expected outcome
Resource resyncing to wait until after the
ack-role-account-map
ConfigMap is read or future resyncs pivots to the cross account role if the ConfigMap is read after resyncing starts.Environment
The text was updated successfully, but these errors were encountered: