Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A potential risk in volcano that could lead to takeover of the cluster #3446

Closed
HouqiyuA opened this issue Apr 28, 2024 · 10 comments
Closed
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug.

Comments

@HouqiyuA
Copy link

Dear Team Members:

Greetings! Our team is very interested in your project and we recently identified a potential RBAC security risk while doing a security assessment of your project. Therefore, we would like to report it to you and provide you with the relevant details so that you can fix and improve it accordingly.

Details:

In this Kubernetes project, there exists a ClusterRole that has been granted list secrets high-risk permissions. These permissions allow the role to list confidential information across the cluster. An attacker could impersonate the ServiceAccount bound to this ClusterRole and use its high-risk permissions to list secrets information across the cluster. By combining the permissions of other roles, an attacker can elevate the privileges and further take over the entire cluster.

We construct the following attack vectors to address the above risks:

First, you need to get the Token for the ServiceAccount that has this high-risk privilege. if you are already in a Pod with this over-privileged privilege, you can run the following command directly to get the Token: cat /var/run/secrets/kubernetes.io/serviceaccount/ token. If you are on the node and not in the Pod, you can run the following command to get the kubectl describe secret .

Use the obtained Token information to authenticate with the API Server. By including the Token in the request, you can be recognized as a legitimate user with the ServiceAccount and gain all privileges associated with the ServiceAccount. This ServiceAccount identity can therefore be used to list all the Secrets in the cluster.

We give two ways to further utilize ServiceAccount Token with other permissions to take over the cluster:

Method 1: Elevation of Privileges using ServiceAccount Token bound to ClusterAdmin.

Use a Token with the privileges of the ClusterAdmin role that has the authority to control the entire cluster. By authenticating with this Token, you can gain full control of the cluster.

Method 2: Create privileged containers using the ServiceAccount Token with the create pods permission. If you have a ServiceAccount with create pods privileges in your k8s cluster, you can use the Token in the Secrets of this ServiceAccount to create a privileged container that mounts the root directory. This container is excluded from the master node by setting a taint tolerance, which leaks the master's kubeconfig configuration file. In this way, the attacker can take over the entire cluster.

For the above attack chain we have developed exploit code and uploaded it to github: https://github.com/HouqiyuA/k8s-rbac-poc

Mitigation methods are explored:

Carefully evaluate the permissions required for each user or service account to ensure that it is following the principle of least privilege and to avoid over-authorization.

If list secrets is a required permission, consider using more granular RBAC rules. Role Binding can be used to grant list secrets permissions instead of ClusterRole, which restricts permissions to specific namespaces or resources rather than the entire cluster.

Isolate different applications into different namespaces and use namespace-level RBAC rules to restrict access. This reduces the risk of privilege leakage across namespaces

Looking forward to hearing from you and discussing this risk in more detail with us, thank you very much for your time and attention.

Best wishes.

HouqiyuA

@HouqiyuA HouqiyuA added the kind/bug Categorizes issue or PR as related to a bug. label Apr 28, 2024
@Monokaix
Copy link
Member

Glad to receive your feedback! Volcano uses a standard serviceAccount & RBAC rule to grant pod permissions, and seems that k8s has some measures to avoid serviceaccount permission issue like https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection, is this can satisfy your need?

@HouqiyuA
Copy link
Author

HouqiyuA commented Apr 29, 2024 via email

@Monokaix
Copy link
Member

I found a ClusterRole named "Volcanic-controllers" found in the volcano project code, which has the list secrets permission. Is granting clusterRole list secrets permission necessary in this project? Marty Mcfly @.> 于2024年4月29日周一 10:43写道:

Glad to receive your feedback! Volcano uses a standard serviceAccount & RBAC rule to grant pod permissions, and seems that k8s has some measures to avoid serviceaccount permission issue like https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#serviceaccount-token-volume-projection, is this can satisfy your need? — Reply to this email directly, view it on GitHub <#3446 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBBY3LTWUNUPSTSLULLRGO3Y7WXTNAVCNFSM6AAAAABG4Z6BTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRG44TQNRTHE . You are receiving this because you authored the thread.Message ID: @.
>

Seems that volcano controller only used GET and CREATE verbs currently: )

@Monokaix
Copy link
Member

Maybe we can remove list permission of volcano controller.

@Monokaix
Copy link
Member

/good-first-issue

@volcano-sh-bot
Copy link
Contributor

@Monokaix:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@volcano-sh-bot volcano-sh-bot added good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Apr 29, 2024
@HouqiyuA
Copy link
Author

HouqiyuA commented Apr 29, 2024 via email

@Monokaix
Copy link
Member

Monokaix commented May 8, 2024

fixed by #3449

@Monokaix
Copy link
Member

Monokaix commented May 8, 2024

/close

@volcano-sh-bot
Copy link
Contributor

@Monokaix: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants