Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorrectly parsing arrays of stringvals #2574

Open
mwright-pivotal opened this issue Aug 16, 2024 · 4 comments
Open

Incorrectly parsing arrays of stringvals #2574

mwright-pivotal opened this issue Aug 16, 2024 · 4 comments

Comments

@mwright-pivotal
Copy link

Terraform version: v1.9.4
Kubernetes provider version: 2.32.0
Kubernetes version: v1.29.6+aba1e8d

Given:

    resource "kubernetes_manifest" "rook-operator-openshift" {
  for_each = { for k, v in provider::kubernetes::manifest_decode_multi(file("${path.module}/infra/rook/operator-openshift.yaml")) : k => v }
  manifest = each.value
  field_manager {
    # set the name of the field manager
    name = "myteam"

    # force field manager conflicts to be overridden
    force_conflicts = true
  }
}

and YAML source found here: https://github.com/mwright-pivotal/platform_automation_hcp_rhos/blob/main/infra/rook/operator-openshift.yaml
which ultimately came from: https://github.com/rook/rook/blob/master/deploy/examples/operator-openshift.yaml

Results in:

Debug Output:

{"@level":"error","@message":"Error: Provider produced inconsistent result after apply","@module":"terraform.ui","@timestamp":"2024-08-15T14:21:50.677977Z","diagnostic":{"severity":"error","summary":"Provider produced inconsistent result after apply","detail":"When applying changes to kubernetes_manifest.rook-operator-openshift[\"1\"], provider \"provider[\\\"registry.terraform.io/hashicorp/kubernetes\\\"]\" produced an unexpected new value: .object.volumes[1]: was cty.StringVal(\"projected\"), but now cty.StringVal(\"emptyDir\").\n\nThis is a bug in the provider, which should be reported in the provider's own issue tracker."},"type":"diagnostic"}
{"@level":"error","@message":"Error: Provider produced inconsistent result after apply","@module":"terraform.ui","@timestamp":"2024-08-15T14:21:50.678179Z","diagnostic":{"severity":"error","summary":"Provider produced inconsistent result after apply","detail":"When applying changes to kubernetes_manifest.rook-operator-openshift[\"1\"], provider \"provider[\\\"registry.terraform.io/hashicorp/kubernetes\\\"]\" produced an unexpected new value: .object.volumes[2]: was cty.StringVal(\"emptyDir\"), but now cty.StringVal(\"hostPath\").\n\nThis is a bug in the provider, which should be reported in the provider's own issue tracker."},"type":"diagnostic"}
{"@level":"error","@message":"Error: Provider produced inconsistent result after apply","@module":"terraform.ui","@timestamp":"2024-08-15T14:21:50.678195Z","diagnostic":{"severity":"error","summary":"Provider produced inconsistent result after apply","detail":"When applying changes to kubernetes_manifest.rook-operator-openshift[\"1\"], provider \"provider[\\\"registry.terraform.io/hashicorp/kubernetes\\\"]\" produced an unexpected new value: .object.volumes[3]: was cty.StringVal(\"hostPath\"), but now cty.StringVal(\"projected\").\n\nThis is a bug in the provider, which should be reported in the provider's own issue tracker."},"type":"diagnostic"}

Steps to Reproduce

Install Rook Ceph using instructions documented here: https://github.com/rook/rook/blob/master/Documentation/Getting-Started/ceph-openshift.md using the terraform kubernetes provider.

Note: the test does not likely requirement openshift kubernetes to reproduce this error if the test is isolated to simply deploying the operator yaml and removing extraneous references, like service accounts etc. Since the error is not coming from kubernetes itself and seems to be coming from the provider parsing logic, successful deployment to kubernetes is not directly a requirement for resolution.

Expected Behavior

operator-openshift.yaml is comprised of many YAML objects. kubernetes_manifest.rook-operator-openshift["0"] and kubernetes_manifest.rook-operator-openshift["1"] both have an attribute named volumes and each have a list of string values. kubernetes_manifest.rook-operator-openshift["0"] and kubernetes_manifest.rook-operator-openshift["1"] should both be parsed successfully.

Actual Behavior

kubernetes_manifest.rook-operator-openshift["0"] is successfully parsed, yet kubernetes_manifest.rook-operator-openshift["1"] is not.

Important Factoids

It appears the provider parsing is somehow conflating values for volumes from kubernetes_manifest.rook-operator-openshift["0"] and kubernetes_manifest.rook-operator-openshift["1"]

@appilon appilon removed their assignment Aug 16, 2024
@alexsomesan
Copy link
Member

This might actually be a side-effect of provider::kubernetes::manifest_decode_multi() function while transcoding YAML. We will have a look into it.

Regardless, we do not recommend this approach of on-the-fly transcoding from YAML documents because it discards a lot of the benefits that Terraform brings over plain YAML, such as attribute value interpolation and establishing a dependency order between resources, etc.

Instead we advise to do the transcoding in advance and save the results as *.tf files with full-fledged terraform resources. That way, the source configuration is stable and the parsing order of such types as lists and maps is reproducible.

@mwright-pivotal
Copy link
Author

mwright-pivotal commented Aug 26, 2024 via email

@alexsomesan
Copy link
Member

Yes, that's correct. I'm suggesting to do the conversion in a prior step. That's just best practice, IMO.

You could either use Terraform itself with the built-in provider function or this simpler tool that makes it a bit easier: https://github.com/jrhouston/tfk8s (author is a member of our team).

The tfk8s tool also does bulk conversion of entire repos which you could then push into version control and consume as a module.

@mwright-pivotal
Copy link
Author

Quick update... I generated https://github.com/mwright-pivotal/platform_automation_hcp_rhos/blob/main/infra/rook/operator-openshift.tf using tfk8s. It also generates the same error when resources are separated into individual kubernetes_manifests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants