[RFE] Add new "InheritedClusterRoles" field

[samjustus]

related to Global Roles v2

SURE-6636

[MbolotSuse]

Validation Template

What was fixed, or what change have occurred

New field

A new field has been added, inheritedClusterRoles to the GlobalRoles resource. This field lists the names of roleTemplates that this GlobalRole grants in all downstream clusters.

Controllers

The controllers for this field function in the following way:

• When a GlobalRoleBinding is made, it lists all current CRTBs which are owned by this GRB
  • A CRTB is owned by a GRB if the authz.management.cattle.io/grb-owner label is set to the GRB's name. (For example, CRTB example is owned by grb test-grb if the CRTB has the label authz.management.cattle.io/grb-owner set to test-grb)
• For each cluster, the following process is followed:
  • All invalid CRTBs owned by this GRB are removed.
    • In short, an owned CRTB is considered invalid if it targets a different user/group than the grb, is being deleted, targets a roleTemplate not requested by the GRB, or is for the local cluster.
    • In detail, A CRTB is considered invalid for a given GRB if any one of the following is true:
      • ClusterName is local
      • The subject (userName/groupPrincipalName) is for a different subject than the GRB
      • The RoleTemplateName is for a name not listed in the GRB's InheritedClusterRoles field
      • The CRTB is being deleted
      • There exists another valid CRTB for the same RoleTemplateName
  • All missing roleTemplates (roleTemplates whose names are in InheritedClusterRoles but there is no valid CRTB for) have a single CRTB created for them, targeting this cluster and the subject specified in the GRB.
    • For each CRTB, the authz.management.cattle.io/grb-owner label is set to the GRB that we are processing.
    • For each CRTB, an owner reference is created tying that CRTB back to the GRB that we are processing (separate from the label).
    • For each CRTB, a label is added indicating that the CRTB is managed by Rancher.

Enqueuers

In addition, several objects have been setup that trigger changes in related objects, to ensure consistency when something other than a GlobalRoleBinding Changes.

• When a GlobalRole that inherits at least one clusterRole is changed, all GRBs which give permissions on that GlobalRole are enqueued and run through the above process.
  • This is to ensure that when adding/removing a new InheritedClusterRole, those permissions are granted to all active bindings.
• When a cluster is added, all GlobalRoles that inherit at least on clusterRole are enqueued. This, in turn, enqueues all GRBs which reference those globalRoles.
  • This is to ensure that when a new cluster is added, permissions are synced to that cluster too.
  • An annotation is added to the cluster after it has been enqueued which makes sure this "initialSync" only happens once, since it enqueues all globalRoles.
• When a CRTB that is owned by a GRB is changed, the owning GRB is enqueued.
  • This is to prevent users from removing/corrupting CRTBs that are created as part of this functionality.

Webhook Validations

New webhook checks have been added for additional security on this field. These checks are documented as part of our webhook docs, but a summary is included here for convienence.

Global Roles

• When creating a new GlobalRole, a roleTemplate which violates one of the following criteria (or doesn't exist) can't be used:
  • Context is anything but cluster
  • Locked is true
  • This prevents users from using a project-type roleTemplate or a locked RoleTemplate.
• Users cannot create/change GlobalRoles which have more permissions in inheritedClusterRoles then they possess
  • This bucket is separate from the Rules bucket. In short, users who have get on pods through Rules can't give get on pods through InheritedClusterRoles - you need to have the permission through that field specifically.
    • This is done to clearly separate local vs remote permissions.
    • This check does not apply to users who have a GRB to Admin or Restricted Admin
  • This check does not affect changes which only change the metadata (i.e. add a label or annotation)
    • This exception was made to allow Rancher's legacy controllers to properly work with this check

Global Role Bindings

• When creating a new GlobalRoleBinding, the GlobalRole that it refers to can't refer to a RoleTemplate which violates one of the following criteria (or doesn't exist):
  • Context is anything but cluster
  • Locked is true
  • This prevents users from using a project-type roleTemplate or a locked RoleTemplate.
• Users cannot create/change GlobalRoleBindings where the GlobalRole they give has more permissions in inheritedClusterRoles then they possess
  • This bucket is separate from the Rules bucket. In short, users who have get on pods through Rules can't give get on pods through InheritedClusterRoles - you need to have the permission through that field specifically.
    • This is done to clearly separate local vs remote permissions.
    • This check does not apply to users who have a GRB to Admin or Restricted Admin
  • This check does not affect changes which only change the metadata (i.e. add a label or annotation)
    • This exception was made to allow Rancher's legacy controllers to properly work with this check

ClusterRoleTemplateBindings

• The GRB ownership label authz.management.cattle.io/grb-owner has the following requirements:
  • The value can't be changed or removed.
  • On create, the value must refer to a valid GRB (i.e. one that exists)

RoleTemplates

• RoleTemplates which are inherited by one or more GlobalRoles can't be deleted

Areas or cases that should be tested

Basic Scenarios

1. Create a GlobalRole with inheritedClusterRoles, bind a user to the GlobalRole, ensure that the user has CRTBs created for all downstream clusters (but not local).
2. Remove the GlobalRoleBinding created in 1. Ensure that the backing CRTBs are removed and that the user loses access.
3. Re-bind the user to the GlobalRole created in 1. Add a new RoleTemplate with different permissions to inheritedClusterRoles. Ensure that the backing CRTBs were made for the new RoleTemplate as well.
4. Remove the new RoleTemplate from the GlobalRole's inheritedClusterRoles. Ensure that the backing CRTBs for the RoleTemplate added in 3 were removed, but the CRTBs created for the RoleTemplate added in 1 remain.
5. Add a new cluster, ensure that there was a new CRTB for the RoleTemplate used in 1.
6. As an admin user, ensure that you can create a GlobalRole with cluster-owner as an inherited cluster role, and that you can bind a user to this role.
7. As a restricted admin user, ensure that you can create a GlobalRole with cluster-owner as an inherited cluster role, and that you can bind a user to this role.
8. Delete the GlobalRole created in 1. Ensure that all GlobalRoleBindings referencing this role as removed, and that the CRTBs owned by those GRBs are removed.

Advanced/Corruption Scenarios

1. Create a GlobalRole with inheritedClusterRoles, bind a user to the GlobalRole, ensure that the user has CRTBs created for all downstream clusters (but not local).
2. Attempt to remove the authz.management.cattle.io/grb-owner label from a backing CRTB, ensure that this fails.
3. Delete the backing CRTB, ensure that Rancher automatically creates another one.
4. Add a finalizer to the backing CRTB and delete it, ensure that Rancher automatically creates another one.
5. Create a CRTB which claims ownership by the GRB but refers to a different RoleTemplate than the one specified in the GRB, ensure Rancher deletes it.
6. Create a CRTB which claims ownership by the GRB but refers to a different user than the one specified in the GRB, ensure Rancher deletes it.
7. Create a CRTB which claims ownership by the GRB but refers to a different group than the one specified in the GRB, ensure Rancher deletes it.
8. Create a CRTB which claims ownership by the GRB but targets the local cluster, ensure Rancher deletes it.
9. Create a CRTB which claims ownership by the GRB but refers to the same RoleTemplate (and same subject, same cluster, not deleting) as an existing CRTB, ensure Rancher deletes it.
10. Create a custom RoleTemplate with a context of project. Try to create a GlobalRole that lists this role as an InheritedClusterRole. Ensure that this fails.
11. Create a custom RoleTemplate which is locked. Try to create a GlobalRole that lists this role as an InheritedClusterRole. Ensure that this fails.
12. Create a GlobalRole with a valid RoleTemplate. Update the RoleTemplate to be locked. Validate that you can add a new, not-locked roleTemplate to the GlobalRole's inheritedClusterRoles.
13. Create a GlobalRole with a valid RoleTemplate. Update the RoleTemplate to be locked. Validate that you can't bind a user to the GlobalRole.
14. Create a valid custom RoleTemplate. Create a GlobalRole which uses this RoleTemplate. Try to delete the RoleTemplate, validate that you can't.

Upgrade Scenarios

1. Upgrade rancher to 2.8-head. Create a new GlobalRole which uses inheritedClusterRoles. Rollback to the earlier version. Validate that the existing permissions remain, but that no new permissions are granted, and no errors are in the log (not sure how valid this is).
2. Upgrade rancher to 2.8-head. Bind a user to admin, and another user to restricted admin. Rollback to the earlier version. Validate the admin/restricted admin permissions still works (make a new cluster, ensure the permissions are synced to that cluster, review the logs for errors).

What areas could experience regressions

Performance - this change can result in new processing time for existing global roles and associated bindings, even if they don't use inheritedClusterRoles. This could be particularly pronounced on the creation on new clusters.

Upgrade/Rollback - this introduces a new field, and as such rollback may experience some issues.

Admin/Restricted Admin - Users with these GlobalRoles need to be able to create GlobalRoles with any inheritedClusterRole.

Feature Example

Creating a GlobalRole can be done by putting the following in gr.yaml

apiVersion: management.cattle.io/v3
kind: GlobalRole
displayName: Example Role
description: Base user + Read-only on all downstream clusters
metadata:
  name: example-gr
inheritedClusterRoles:
  - projects-view
rules:
- apiGroups:
  - management.cattle.io
  resources:
  - preferences
  verbs:
  - '*'
- apiGroups:
  - management.cattle.io
  resources:
  - settings
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - management.cattle.io
  resources:
  - features
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - project.cattle.io
  resources:
  - sourcecodecredentials
  verbs:
  - '*'
- apiGroups:
  - project.cattle.io
  resources:
  - sourcecoderepositories
  verbs:
  - '*'
- apiGroups:
  - management.cattle.io
  resources:
  - rancherusernotifications
  verbs:
  - get
  - list
  - watch

You can then create the GlobalRole using kubectl:

kubectl create -f gr.yaml

Users can be bound to this through the UI, or by making a globalRole by putting the following in grb.yaml:

apiVersion: management.cattle.io/v3
kind: GlobalRoleBinding
metadata:
  name: example-grb
userName: u-123xyz
globalRoleName: example-gr

You can then create the GlobalRoleBinding using kubectl:

kubectl create -f grb.yaml

[Priyashetty17]

Update: Most of the tests for this have been completed. @anupama2501 is executing the performance and backup/restore checks. Once those are done, we'll mark this issue as "Done".

[anupama2501]

Verified the rancher backup/restore test cases:
Following cases were verified:
More details steps were provided in QASE.

1. Rancher upgrade and roll back using backup/restore charts v2.7.7 >> v2.8.0 >> v2.7.7 with migration
2. Rancher upgrade and roll back using backup/restore charts v2.7.7 >> v2.8.0 >> v2.7.7 without migration
3. Rancher migrate to a different server using backup/restore operator

Steps for test 3

• Install rancher on 2.8.0-alpha2 - rancher1
• Install backup/restore operator
• Create a global role includes InheritedClusterRoles cluster owner - grole1
• As an admin, create 2 downstream clusters - RKE1 and RKE2 node driver. Verify the clusters are active.
• As an admin, create a few users:
• u1 with the global roles user-base and custom grole1
• u2 with the global role standard-user and custom grole1
• u3 with the standard user role
• u4 with restricted admin role
• Verify the following are created for users u1 and u2 in both the downstream clusters:
• Verify the global role bindings are created in the local clusters
• Verify the global role is created in the local clusters
• Verify the clusterroletemplate bindings are created in local cluster for all downstream clusters
• Verify the role bindings are created in local cluster and the downstream clusters for all downstream clusters
• Verify the cluster role bindings are created in local cluster and the downstream clusters for all downstream clusters
• Take a backup from local cluster explorer >> backups. Lets call it backup1
• Create a new rke standalone cluster
• Install backup/restore operator version same from 2
• Bring down the nodes of rancher1 server
• Restore the backup taken in step12
• Install rancher version 2.8.0-alpha2
• Verify the installation goes through and no errors observed
• Create a new user - u5 with standard user role and custom global role - grole1
• Verify the user is added in both the downstream clusters as cluster owner

[anupama2501]

Verified performance checks on v2.8.0-alpha2:

1. Create a large number of users on rancher v2.8.0 - 3k
2. Create a custom global role with cluster-member inheritedClusterRole permissions
3. Create a downstream cluster - rke1
4. Assign it to the 3k users
5. Verify the CRTBs are created for all the users in the downstream cluster
   

Performance check: Verified the rolebindings count for 1000 users with global roles v2, inheritedClusterRole=ClusterOwner permissions are 8838 compared to 20174 for 1000 restricted admin users.

[anupama2501]

Reopening for 1 more validation on performance checks.

[anupama2501]

Updated the above comment with results. Closing the issue as validations look good.

[MbolotSuse]

Release Note:

Global Roles have a new optional field ( inheritedClusterRoles) which grants a cluster-level permissions on all downstream clusters (meaning all clusters beside the local cluster).