-
Notifications
You must be signed in to change notification settings - Fork 969
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to use kubernetes provider with fixed limited permissions - see here: https://github.com/hashicorp/terraform-provider-azurerm/pull/21229 #2072
Comments
Tagging @browley86 here, I think this time its a kubernetes-provider issue. |
I guess this also is the fact for the helm provider. I link the issues: hashicorp/terraform-provider-helm#1114 |
Hey @slzmruepp, I actually think there is a way to get this working but the setup has to be done at the provider level. There is a post about this where the user uses the Exec plugins that the k8s provider exposes. I haven't had time to give it a go myself but that was my plan initially after getting the kubeconfig going using the new API endpoint that was just released. I'll try and get this going in the next few days. |
Ok so I got it to work, there is good news and bad news. The bad news is that, giving the permissions of the Service Principals, it cannot read the required
Which is fed into the kubelogin:
My Service Principal, out of the box, doesn't have the rights to lookup that user. Instead, because I had the server id in my own
There may be another way to get the app id from the initial data source but I did not see an option on an admittedly quick scan. The vault workaround is hacky but good enough for me in the short term. Hope that helps. |
Thank you, but I consider as a workaround. This is not as documentation suggests. Is there an effort from hashicorp to solve this in the provider code? Also, when the data structure in tf fetches the kubeconf from the listClusterUserCredential Endpoint, it gets back the kubeconf as a base64 encoded string. If you look at my post where I show the decoded kubeconf, this kubeconf contains the server-id. Is there a way to use the azcli token to authenticate to the cluster in the background? But, I would suggest here this is a bug. First, the documentation implicates that it works OOB like this. Second, there is no documentation about kubelogin use. I am talking about a sole CI/CD approach. We never run TF locally, only in the pipelines. So the pipeline agents would have kubelogin available. Isnt it enough to reformat the kubeconf in the background to use azcli? Basically what kubelogin does? Because when you run tf in pipeline, an azcli token must be existing somehow anyway, right? From: users:
- name: clusterUser_XXX
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- get-token
- --environment
- AzurePublicCloud
- --server-id
- XXX
- --client-id
- XXX
- --tenant-id
- XXX
- --login
- devicecode
command: kubelogin
env: null
provideClusterInfo: false to: users:
- name: clusterUser_XXX
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- get-token
- --login
- azurecli
- --server-id
- XXX
command: kubelogin
env: null
provideClusterInfo: false Thanks |
@slzmruepp - so uh, I did it. I didn't like it. But yeah:
That will get it without my hack of hard-coding it in a different backend. Some context: I had to plan to a file then decode the file to actually see where in the JSON the value showed up. Aside from my references to my vault variable, the only other place it showed up is in the Edit for completeness, here is the final provider block:
|
Haha, yes this is quite a bit. Problem with Azure DevOps Pipelines and the terraform tasks is, there is no access to the sp secret. So this is all done in the background. Maybe one approach would be to fetch the kubeconf and put it on the filesystem, then run kubelogin convert-kubeconfig -l azurecli and in the provider section just refer to this kubeconf file. But how can such things run before provider init? @browley86 thanks for the effort, but I dont think this is a feasible approach because to fetch the service principal keys from a key vault would have been possible all along. But we dont even know the secrets ourself because we create them programmatically and create directly the service connections in azure devops. So, I would suggest there should be a feasable solution from the provider itself, thus I filed this issue. We have kubelogin on our azure agents but how to configure properly is a mess. Also the documentation of the provider is still wrong because it just dont work... I tried this but still get unauthorized when trying reading a data "kubernetes_namespace_v1" object: # Configure the Microsoft Azure Provider
provider "kubernetes" {
host = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.host
username = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.username
password = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.password
client_certificate = base64decode(data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.client_certificate)
client_key = base64decode(data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.client_key)
cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.cluster_ca_certificate)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "kubelogin"
args = [
"get-token",
"--login",
"azurecli",
"--server-id",
"Manually extracted for testing"
]
}
} |
In fairness to those tracking the issue, this is a different problem though: the password for the service principal is exposed via the Service Principal Password which can then be referenced later in the run or, if another team is creating the SP via terraform in a different repo, they would need to put the password in some backend like Hashicorp vault or Azure Key Vault so that your user could pick it up later (it looks like the data lookups for SPs don't have the password). Re: the idea of making the file, it might work using local_file with Edit: just noticed there is local_sensitive_file which is way more appropriate for the kubeconfig, so some quick pseudo-code:
The issue there though is now there's a file with sensitive stuff lying around and would need to be cleaned at the end of every run. |
So the catch22 when I try the following is, that the plan step always fails because the kubeconf is not existing yet, thus our pipelines fail. resource "local_sensitive_file" "kubeconfig" {
content = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config_raw
filename = "./kubeconfig"
provisioner "local-exec" {
command = "kubelogin convert-kubeconfig --login azurecli --kubeconfig ./kubeconfig"
}
}
provider "kubernetes" {
config_path = local_sensitive_file.kubeconfig.filename
} Is there someone from Team Hashicorp watching this? Any Solutions? Thanks! |
Ok so now I got an acceptable solution: provider "kubernetes" {
host = data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.host
cluster_ca_certificate = base64decode(data.azurerm_kubernetes_cluster.aks_provider_config.kube_config.0.cluster_ca_certificate)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "kubelogin"
args = [
"get-token",
"--login",
"azurecli",
"--server-id",
var.env_config[var.ENV][ "server_id" ]
]
}
} So the main issue with my former approach was that I was under the impression that if I dont remove these lines:
from the provider config, it still uses the kubelogin token. BUT IT DONT. IT TRIES TO AUTH WITH THE CERT and KEY. Thats why my first approach failed... So if you run your tf in an azcli token environment this is the way to go... |
Still I think this should be a built in feature of the provider to support such behavior without hacking the exec plugin. |
Hi @slzmruepp according to our documentation this is the correct way to configure the provider for auth plugins. |
@sheneska - could you please provide a link to that documentation? |
©sheneska Obviously this is not the way how it works when you use azurecli context login which a lot of TF Tasks in pipelines do. So either way is not part of the documentation so far. At least I did not find it. So this feels pretty hacky and only some sources on the internet document at least the SPN way of configure the exec plugin of the provider. So if this is the "official way", I would certainly expect this to be documented. See here: All documentation points to a certainly NOT working approach for limited permissions including AKS kubelogin. |
Hi @browley86 we are documenting the use of using exec plugins, however we are not able to document every configuration argument for every plugin that is available. Please refer to the documentation for the specific plugins on how to actual configure them. |
@sheneska, so totally get it, the exec statement is a sort of catch-all for kubernetes plugins. Documenting every use case is impossible. That said, in the case of Azure AKS service, they seem to have at least, for the time being, standardized on kubelogin so documenting that use case would probably be worthwhile. That said, my ask here is less as a bug and more of a feature request: it would be extremely nice/convenient for the data object to expose the server ID for the kubelogin plugin. So, for example, instead of using:
It would be way nicer to use
Considering the terraform provider gets this as part of it's response in the code, it would be nice to be able to expose it as an additional output so the end-user(s) can leverage it without having to hack the |
@browley86 would it make it easier to use something like
and
to lookup those two from the current context rather than having to also look them up from your vault / key_vault ? It does not help with getting the server_id however ;) |
Btw. it looks like the yamldecode is not working anymore or it is not working because of our AKS API setup here (private API server with VNet integration):
|
So my above account got swallowed but the email made its way to my personal account. Anywho, I had the wrong key above, it should be:
|
I am not sure why this extended server_id is required in your context? I got the following code working: # data "azuread_service_principal" "aks" {
# display_name = "Azure Kubernetes Service AAD Server"
# }
provider "kubernetes" {
host = module.aks.cluster.kube_config.0.host
cluster_ca_certificate = base64decode(
module.aks.cluster.kube_config[0].cluster_ca_certificate,
)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "kubelogin"
args = [
"get-token",
"--login",
"azurecli",
"--server-id",
"6dae42f8-4368-4678-94ff-3960e28e3630" # data.azuread_service_principal.aks.client_id
]
}
}
resource "kubernetes_namespace" "default" {
metadata {
name = "helloworld"
}
} I have local accounts disabled and I am using my user account with the "Azure Kubernetes Service RBAC Cluster Admin" role assigned. I would expect that any azure cli authenticated context would work here? |
Sorry for the long delay but I just wanted to close the loop here: this, above, is the best answer. In short, Microsoft creates an Enterprise Application called "Azure Kubernetes Service AAD Server" and the Application ID of that Enterprise app is the
A few notes though: this is all to workaround the fact that some people consider client_ids and object_ids to be sensitive. If this is the case the data lookup with a TLDR: If you are limited to |
Hi everyone! I found this issue looking for solutions and thanks to this last response I was able to understand it and it worked on my case. So I share it here in case it helps others. In our case, we provided all the necessary roles to the Service Principal (to be cluster-admin in Kubernetes), but it was still giving 401 Unauthorized, and this was because I was trying to use a wrong It also blew my mind as it's not clearly documented anywhere in Azure's documentation... But at least it's on kubelogin's documentation. In case it helps, that was the missing piece in our case This is the magic
|
Terraform Version, Provider Version and Kubernetes Version
Affected Resource(s)
Terraform Configuration Files
Debug Output
Steps to Reproduce
See here:
hashicorp/terraform-provider-azurerm#21183
Expected Behavior
Kubernetes resources should be able to be fetched via data and resources should be created according to the limited permissions in the specific namespace
Actual Behavior
Terraform returns with error "unauthorized"
Important Factoids
I did some testing and the outcome is, that the fetch of the limited permissions works now:
The base64 decoded kubeconf (the value) looks correct:
To debug, I tried to use this service principal sequence for kubectl. I use the following sequence:
After the kubelogin convert, the kubeconfig under .kube/config looks like this:
So I don't know what runs behind the terraform curtain, but I suspect the kubelogin part of the steps is not accounted for, thus getting the "unauthorized" response because it does not get the token from the azurecli context.
References
kubernetes_cluster
- support for using the data source with limited permissions terraform-provider-azurerm#21229Community Note
The text was updated successfully, but these errors were encountered: