AWS SAML connection error on AWS ElasticSearch cluster since upgrading #189
Comments
|
Hello, can you please include the following:
|
|
Provider version provider elasticsearch {
url = "https://${aws_elasticsearch_domain.domain.endpoint}"
sign_aws_requests = true
aws_assume_role_arn = "arn:aws:iam::account-id:role/deployment-role-name"
elasticsearch_version = "7.9"
}Some resources I'm trying to manage with the elasticsearch provider
resource elasticsearch_opendistro_ism_policy delete_after_30d {
policy_id = "delete_after_30d"
body = <<EOF
{
"policy": {
"description": "Delete indices older than 30 days",
"default_state": "hot",
"schema_version": 1,
"states": [
{
"name": "hot",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "30d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
]
}
}
EOF
}
resource elasticsearch_index_template fluent_bit {
name = "fluent-bit-template"
body = <<EOF
{
"index_patterns": [
"logstash*"
],
"settings": {
"index": {
"opendistro": {
"index_state_management": {
"policy_id": "${elasticsearch_opendistro_ism_policy.delete_after_30d.id}"
}
}
}
}
}
EOF
}
resource "elasticsearch_opendistro_roles_mapping" "fluent_bit_write" {
role_name = "logstash"
description = "Allow fluent-bit pods to forward logs to ElasticSearch."
backend_roles = [
aws_iam_role.monitoring_fluent_bit_role.arn,
"arn:aws:iam::account-id:role/*",
]
}
resource "elasticsearch_opendistro_roles_mapping" "all_access" {
role_name = "all_access"
description = "Allow all actions."
backend_roles = [
"SSO Group Name",
"arn:aws:iam::account-id:role/AWSReservedSSO_SSO_ROLE_NAME", # AWS SSO provisioned role
"arn:aws:iam::account-id:role/deployment-role-name",
]
}
resource "elasticsearch_opendistro_role" "readall_and_monitor_global" {
role_name = "readall_and_monitor_global"
description = "readall_and_monitor with access to the global tenant"
cluster_permissions = ["cluster_monitor", "cluster_composite_ops_ro"]
index_permissions {
index_patterns = ["*"]
allowed_actions = ["read", "indices_monitor"]
}
tenant_permissions {
tenant_patterns = ["global_tenant"]
allowed_actions = ["kibana_all_read"]
}
}
resource "elasticsearch_opendistro_roles_mapping" "readall_and_monitor_global" {
role_name = "readall_and_monitor_global"
description = "Allow read only and monitor actions on the global tenant."
backend_roles = [
"SSO Group Name",
]
}I didn't find anything particularly interesting in the logs with |
|
I've seen this a lot in the past few days, BUT it's always been a configuration issue on my end. Try applying your authn/authz terraform using tldr: The way it says "you dont have permission" is by saying "no Elasticsearch node available". |
|
I spent some of today stepping through all of this to try and get to the bottom of the issue, and I think # this doesn't work
provider elasticsearch {
url = "https://${aws_elasticsearch_domain.domain.endpoint}"
sign_aws_requests = true
aws_assume_role_arn = "arn:aws:iam::account-id:role/my-deployment-role"
}As mentioned, the above provider config gives me the # this works
provider elasticsearch {
url = "https://${aws_elasticsearch_domain.domain.endpoint}"
sign_aws_requests = true
aws_access_key = "XXXXXXXXXXXXX"
aws_secret_key = "XXXXXXXXXXXXX"
aws_token = "XXXXXXXXXXXXX"
}Perhaps this is similar to #124? I'm using provider version |
|
Also it might be worthwhile to note that I'm using an SSO profile by doing I used the same profile to assume role with, so it's definitely got the right permissions for that. |
|
Hi @raids, sorry to hear that this broken unexpectedly! Thanks for providing details and debugging.
I'm not sure why it would matter, but would you be able to try downgrading terraform? If the terraform version change started the issue, perhaps reverting would fix it?
This definitely points to an issue - have you tried specifying the profile via the Can you confirm what file the profile information is stored in on disk? |
|
Hi @phillbaker - apologies, I haven't had much time to test this further. Some answers to your questions and then some good news(?) below.
I went ahead with the Terraform 1.0 upgrade as a priority, so running again on a downgraded version of Terraform isn't an easy option for me right now.
We can't use
The SSO profile information is stored in The good news is that I think I know where the issue stems from, and it's not from upgrading provider version:
|
phillbaker
changed the title
|
Thanks @raids, the suggestion to set |
|
Done in 72475b5 |
I'm seeing similar behaviour to #183, but it not exactly the same.
I've recently upgraded from tf 0.13 to 1.0. Since upgrading and migrating the state, this provider doesn't seem able to authenticate with a cluster which it previously was managing (
elasticsearch_opendistro_roles and mappings,elasticsearch_index_templates, etc.)terraform plangives me one of these for each elasticsearch provider resource:The cluster had SAML authentication turned out after it was created, but even since then, there have been hundreds of
plans andapplys without issue. I was previously on provider version1.5.1because of a regression blocking use of IAM roles (#149).I remained on provider version
1.5.1after upgrading terraform, and when I saw these errors, I tried the latest version1.5.7and also1.5.0(as referenced in other issues), but it doesn't help.Let me know if I can do anything to help debug / troubleshoot with a cluster in AWS. Thanks!
Edit: Also, perhaps worth noting that the suggestions from #183 to set
sign_aws_requests = falseandinsecure = truedon't help (not that I'd want to set them long term 😁.The text was updated successfully, but these errors were encountered: