Cross-namespace references

[simongottschlag]

Hi!

We’re migrating our Kubernetes service for our customers from Flux v1 to v2 now and I’m starting to understand the new concepts better.

I’ve been thinking about a case that may be an issue for us:

• Each team has their own namespace (team1 & team2)
• GitRepository is created for each team to their own repository in flux-system
• Kustomization / HelmRelease is created in the team namespace (pointing to the GitRepository)

The issue here is that there’s no obvious way of using RBAC to limit what GitRepositories can be used by different namespaces. team1 can in theory point to team2 repository. This should also be the case if the GitRepository is created in the team namespace, if I’m not wrong.

Our issue here is that we are using the Azure DevOps which doesn’t have deploy keys like all other providers. This means we use a key (PAT) for Azure DevOps with very high permissions for all repositories in all projects.

If we placed GitRepository object in the team namespace, they would have access to the PAT. In our case, we use the same PAT for each GitRepository, which means that cross-namespace references actually help us here - but causes a problem that other teams in theory can point to other GitRepositories.

My thought here is that we shouldn’t care about the limitations with Azure DevOps and think about all the other use cases, where team1 would be able to point their Kustomization to team2 GitRepository in team2 namespace.

That shouldn’t be possible, in my opinion - it breaks RBAC.

We are able to solve our use case with opa-gatekeeper limiting a Kustomization / HelmRelease in the team namespace to only use a GitRepository in flux-system namespace with the same name as the namespace. But this feels like a hack and something that shouldn't be needed, as Flux should be secure out of the box and not break RBAC.

What are other peoples thoughts on this? Am I missing something obvious?

Regards,
Simon

[stefanprodan]

I would create a PAT per repository or per tenant, then you can place the pull secret, source and kustomization in the tenant namespace. You would then need a validation webhook that rejects kustomizations and helmreleases that have spec.sourceRef.namespace specified.

[simongottschlag]

@stefanprodan but should we by default break RBAC? Isn't that something that should have a big opt-in button or something?

I see our use case as something that we can solve by validation webhook, but I think this may be a bigger question.

It feels like there should be a better way than going around RBAC here?

[stefanprodan]

@simongottschlag there is no opt-in button for multi-tenancy but there will be one, see #263

In the multi-tenant mode, kustomize-controller will use the specified service account to query the source, if the role binding will not allow access to GitRepositories cross-namespace, then kustomize-controller will error out for Kustomizations with spec.sourceRef.namespace.

[simongottschlag]

@stefanprodan if we removed the ability to reference other namespaces, what functionality would we loose?

What is the praxis around cross-namespace references in the community right now?

I’ve only noted that it has been used in a few projects so far and in those cases they’ve made it possible to read secrets across namespaces breaking every rule in the book.

Wouldn’t it be “safer” for the community that will use Flux not to enable something like this?

[stefanprodan]

I’ve only noted that it has been used in a few projects so far and in those cases they’ve made it possible to read secrets across namespaces breaking every rule in the book.

That's not true, we don't allow cross-namespace refs for secrets. One good example when you would want to reuse the source is when you have a mono-repo with helm charts and plain yamls apps. The HelmReleases and Kustomizations would be created in each app namespace while the source is defined once in flux-system.

[simongottschlag]

@stefanprodan I meant other places I’ve seen cross-namespace references, they have enabled reading secrets from “other teams”. Example: https://docs.solo.io/gloo-edge/latest/reference/api/github.com/solo-io/gloo/projects/gloo/api/v1/options/azure/azure.proto.sk/#upstreamspec

But reusing something would be possible using either duplicating the resource or maybe making a cluster wide resource (like cert-manager and ClusterIssuer).

My opinion is that we should at least think about not enabling cross-namespace references since it may enable users of Flux to easily do something wrong since we don’t respect RBAC as it is built.

[squaremo]

I have to say going around RBAC makes me distinctly uncomfortable too.

Is the motivating use case for cross-namespace references the one @simongottschlag described -- people want to be able to define a GitRepository that can be used, but not seen? It is hard to come up with a solution for this other than putting GitRepository objects with their secrets in a namespace where the tenant can't read them. Perhaps if it's a namespace where the tenant can read/write Kustomizations and HelmReleases, but not read secrets .. but then what if they want to mount a secret in a HelmRelease.

[simongottschlag]

@squaremo my opinion is that Flux should make it easy for the end user to do the right thing. It should be as secure as possible by default and follow the praxis of the community.

We have two scopes for CRDs, namespaced and cluster-scoped. If we use something that is namespaced, the expected behaviour is that it shouldn't be able to cross the namespace boundary. We are now implementing a namespaced resource and making it cluster-scoped. This won't be expected by people using it. I know others do it this way, but why continue that trend making Kubernetes even harder to understand than what it already is?

If it's really important to cross namespaces, then ClusterGitRepository should be used (as an example) instead of GitRepository - making it clear for the person implementing it that everyone will be able to reference it. If we want to lock down the ClusterGitRepository, then something like opa-gatekeeper can be used for that purpose.

I don't see this as a problem for us, since we are implementing opa-gatekeeper and we do understand the Flux concepts and how all of these things are implemented. Not so sure that the wider community will understand the implications right away and not sure what a security audit would say about it in the future.

[squaremo]

ClusterGitRepository strikes me as a better solution than breaking RBAC expectations. It would necessitate some cluster-level analogue to secrets, since most git repositories will need a key or token. Unless the ClusterGitRepository can refer to a secret in a namespace .. though does this bring us back to RBAC considerations. (If not, that's a solution for being able to use a git repository without having access to its access credentials.)

[simongottschlag]

@squaremo Since the users in a namespace would usually only have access to namespaced resources, they should never be able to create a ClusterGitRepository. Referencing a namespaced resource (secret) from a cluster-scoped resource (ClusterGitRepository) does seem like the logical way of handling it, and without having tested it, it seems like this is the way cert-manager does it with their ClusterIssuer (but I haven't read up on it in detail, don't quote me on it).

Namespaced resources should in my opinion not be able to reference resources outside of the namespace other than cluster-scoped resources.

[squaremo]

The proposed "security model for impersonation" assumes cross-namespace references to sources are kept, but restricts their use by requiring everything using them to impersonate a user local to its namespace (Stefan also mentioned this idea in a comment, above, I believe). So, if your GitRepository is in flux-system, but you want a Kustomization that refers to it, your Kustomization must be given a local user that has access to the GitRepository in flux-system.

This mitigates some of my unease around subverting RBAC. I think the strongest argument for cross-namespace references is that it means you can grant access to a source without giving away the secrets it uses. There's other ways of doing this -- the Cluster idea is one of them, having some kind of bindings (like RoleBindings) granting access to each namespace is another -- but using cross-namespace references is the simplest, mechanically.

@simongottschlag See what you think of the impersonation proposal linked above, if you get a chance.

[simongottschlag]

@squaremo I still think we should reconsider this. Taking a namespaced resource and making it cluser scoped is a bad practice and shouldn't be encouraged. I think other projects that do this is setting a bad precedence and if we do the same with Flux others will follow.

We should make it easy to use the project, not harder. It should be secure by default and easy to understand, this impersonation makes it quite hard to understand the flow and non-intuitive.

I'm glad it's possible to make it secure, just think we should make sure that we do it the right way.

[stealthybox]

This is how this could be accomplished with #582's RBAC/impersonation design /w cross-namespace Sources.
The proposed default behavior is secure/deny-all.

Here's how you could opt-in to cross-namespace behavior for specific GitRepos:

Team Repos managed in flux-system:

kind: GitRepository
metadata:
  name: team1-app
  namespace: flux-system
spec:
  url: https://git.azuredevops.com/myorg/team1-app
  secretRef:
    name: myorg-azuerdevops-pat
---
kind: GitRepository
metadata:
  name: team2-app
  namespace: flux-system
spec:
  url: https://git.azuredevops.com/myorg/team2-app
  secretRef:
    name: myorg-azuerdevops-pat

RoleBindings for each team's FluxUser to only access their specific repo:

kind: Role
metadata:
  name: view-gitrepo-team1-app
  namespace: flux-system
rules:
  - apiGroups: ["source.toolkit.fluxcd.io/v1beta1"]
    resources: ["gitrepository"]
    verbs: ["get", "watch", "list"]
    resourceNames: ["team1-app"]
---
kind: RoleBinding
metadata:
  name: flux-team1-repo-access
  namespace: flux-system
roleRef:
  kind: Role
  name: view-gitrepo-team1-app
subjects:
  - kind: User
    name: flux:user:team1:reconciler  # default FluxUser from team1 Namespace
---
kind: Role
metadata:
  name: view-gitrepo-team2-app
  namespace: flux-system
rules:
  - apiGroups: ["source.toolkit.fluxcd.io/v1beta1"]
    resources: ["gitrepository"]
    verbs: ["get", "watch", "list"]
    resourceNames: ["team2-app"]
---
kind: RoleBinding
metadata:
  name: flux-team2-repo-access
  namespace: flux-system
roleRef:
  kind: Role
  name: view-gitrepo-team2-app
subjects:
  - kind: User
    name: flux:user:team2:reconciler  # default FluxUser from team2 Namespace

Kustomizations in each namespace referencing team specific repos:

kind: Kustomization
metadata:
  name: app
  namespace: team1
spec:
  path: ./config
  sourceRef:
    kind: GitRepository
    name: team1-app
    namespace: flux-system
---
kind: Kustomization
metadata:
  name: app
  namespace: team2
spec:
  path: ./config
  sourceRef:
    kind: GitRepository
    name: team2-app
    namespace: flux-system

In this setup, neither team has access to the Azure PAT (privileged access to close all repos).

This setup with the central PAT is nice -- it demonstrates how Flux can still be used to create a central approval flow on a control-repo, using native RBAC.
Individual teams could still request access to their own repos via PullRequest which is a self-service element.

You could avoid this RBAC setup by giving each team their own PAT in their own Namespace with the constrained access rights.
The approval flow in that case moves to managing the PAT's and their repoa access.