kubernetes_manifest for external-secrets.io/v1beta1/SecretStore: Plugin did not respond / plugin exited

[aristosvo]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.8.2
Kubernetes provider version: v2.31.0
Kubernetes version: 1.27.x

Affected Resource(s)

• ExternalSecrets/v1beta1 SecretStore via kubernetes_manifest

Terraform Configuration Files

resource "kubernetes_manifest" "secretstore_aws_secretsmanager" {
  manifest = {
    "apiVersion" = "external-secrets.io/v1beta1"
    "kind"       = "SecretStore"
    "metadata" = {
      "name"      = "default-secretstore"
      "namespace" = var.namespace
    }
    "spec" = {
      "provider" = {
        "aws" = {
          "auth" = {
            "secretRef" = {
              "accessKeyIDSecretRef" = {
                "key"  = "key"
                "name" = local.secret_name
              }
              "secretAccessKeySecretRef" = {
                "key"  = "secret"
                "name" = local.secret_name
              }
            }
          }
          "region"  = data.aws_region.current.name
          "role"    = aws_iam_role.external_secret_operator.arn
          "service" = "SecretsManager"
        }
      }
    }
  }
}

Log output

╷
│ Error: Plugin did not respond
│ 
│   with module.secret_management["x-a"].kubernetes_manifest.secretstore_aws_secretsmanager,
│   on ../../../modules/secrets-manager/main.tf line 107, in resource "kubernetes_manifest" "secretstore_aws_secretsmanager":
│  107: resource "kubernetes_manifest" "secretstore_aws_secretsmanager" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.secret_management["x-d"].kubernetes_manifest.secretstore_aws_secretsmanager,
│   on ../../../modules/secrets-manager/main.tf line 107, in resource "kubernetes_manifest" "secretstore_aws_secretsmanager":
│  107: resource "kubernetes_manifest" "secretstore_aws_secretsmanager" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may
│ contain more details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.secret_management["x-t"].kubernetes_manifest.secretstore_aws_secretsmanager,
│   on ../../../modules/secrets-manager/main.tf line 107, in resource "kubernetes_manifest" "secretstore_aws_secretsmanager":
│  107: resource "kubernetes_manifest" "secretstore_aws_secretsmanager" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).UpgradeResourceState call. The plugin logs may
│ contain more details.
╵
2024-07-15T11:18:51.504Z [DEBUG] provider: plugin exited

Sometimes it is erroring out on ReadResource calls already, but always on the same calls in one run:

╷
│ Error: Plugin did not respond
│ 
│   with module.secret_management["x-d"].kubernetes_manifest.secretstore_aws_secretsmanager,
│   on ../../../modules/secrets-manager/main.tf line 107, in resource "kubernetes_manifest" "secretstore_aws_secretsmanager":
│  107: resource "kubernetes_manifest" "secretstore_aws_secretsmanager" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
│ details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.secret_management["x-t"].kubernetes_manifest.secretstore_aws_secretsmanager,
│   on ../../../modules/secrets-manager/main.tf line 107, in resource "kubernetes_manifest" "secretstore_aws_secretsmanager":
│  107: resource "kubernetes_manifest" "secretstore_aws_secretsmanager" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
│ details.
╵
╷
│ Error: Plugin did not respond
│ 
│   with module.secret_management["x-a"].kubernetes_manifest.secretstore_aws_secretsmanager,
│   on ../../../modules/secrets-manager/main.tf line 107, in resource "kubernetes_manifest" "secretstore_aws_secretsmanager":
│  107: resource "kubernetes_manifest" "secretstore_aws_secretsmanager" {
│ 
│ The plugin encountered an error, and failed to respond to the
│ plugin.(*GRPCProvider).ReadResource call. The plugin logs may contain more
│ details.
╵

Debug Output

I was caught by surprise that this error only generates a Plugin did not respond answer.

2024-07-15T11:18:28.471Z [TRACE] provider.terraform-provider-aws_v5.58.0_x5: Served request: @caller=github.com/hashicorp/terraform-plugin-go@v0.23.0/tfprotov5/tf5server/server.go:843 tf_proto_version=5.6 tf_provider_addr=registry.terraform.io/hashicorp/aws tf_rpc=PlanResourceChange @module=sdk.proto tf_req_id=xxx tf_resource_type=aws_route53_zone_association timestamp=2024-07-15T11:18:28.471Z
2024-07-15T11:18:28.472Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-07-15T11:18:28.483Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.58.0/linux_amd64/terraform-provider-aws_v5.58.0_x5 pid=412
2024-07-15T11:18:28.483Z [DEBUG] provider: plugin exited
2024-07-15T11:18:46.896Z [DEBUG] provider.terraform-provider-kubernetes_v2.31.0_x5: Sending HTTP Request: tf_http_op_type=request tf_http_req_body="" tf_http_req_method=GET tf_http_req_version=HTTP/1.1 Authorization="Bearer [MASKED]" new_logger_warning="This log was generated by a subsystem logger that wasn't created before being used. Use tflog.NewSubsystem to create this logger before it is used." @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/logging/logging_http_transport.go:160 Accept=application/json Accept-Encoding=gzip User-Agent="terraform-provider-kubernetes_v2.31.0_x5/v0.0.0 (linux/amd64) kubernetes/$Format" tf_http_req_uri=/apis/apiextensions.k8s.io/v1/customresourcedefinitions @module="kubernetes.Kubernetes API" Host=api.xxx.openshiftapps.com:6443 tf_http_trans_id=xxx timestamp=2024-07-15T11:18:46.896Z
2024-07-15T11:18:47.082Z [DEBUG] provider.terraform-provider-kubernetes_v2.31.0_x5: Sending HTTP Request: Authorization="Bearer [MASKED]" User-Agent="terraform-provider-kubernetes_v2.31.0_x5/v0.0.0 (linux/amd64) kubernetes/$Format" tf_http_req_uri=/apis/apiextensions.k8s.io/v1/customresourcedefinitions tf_http_trans_id=xxx Accept-Encoding=gzip Host=api.xxx.openshiftapps.com:6443 new_logger_warning="This log was generated by a subsystem logger that wasn't created before being used. Use tflog.NewSubsystem to create this logger before it is used." tf_http_op_type=request tf_http_req_version=HTTP/1.1 @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/logging/logging_http_transport.go:160 @module="kubernetes.Kubernetes API" Accept=application/json tf_http_req_body="" tf_http_req_method=GET timestamp=2024-07-15T11:18:47.082Z
2024-07-15T11:18:47.323Z [DEBUG] provider.terraform-provider-kubernetes_v2.31.0_x5: Sending HTTP Request: @caller=github.com/hashicorp/terraform-plugin-sdk/v2@v2.34.0/helper/logging/logging_http_transport.go:160 Accept=application/json Accept-Encoding=gzip tf_http_op_type=request tf_http_req_version=HTTP/1.1 Authorization="Bearer [MASKED]" Host=api.xxx.openshiftapps.com:6443 tf_http_req_body="" tf_http_req_method=GET @module="kubernetes.Kubernetes API" User-Agent="terraform-provider-kubernetes_v2.31.0_x5/v0.0.0 (linux/amd64) kubernetes/$Format" new_logger_warning="This log was generated by a subsystem logger that wasn't created before being used. Use tflog.NewSubsystem to create this logger before it is used." tf_http_req_uri=/apis/apiextensions.k8s.io/v1/customresourcedefinitions tf_http_trans_id=xxx timestamp=2024-07-15T11:18:47.323Z
2024-07-15T11:18:51.187Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/kubernetes/2.31.0/linux_amd64/terraform-provider-kubernetes_v2.31.0_x5 pid=303 error="signal: killed"
2024-07-15T11:18:51.187Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"

Steps to Reproduce

Unsure, for certain runs this is the result while in other situations it works.

• We have a cluster with a lot of CRDs, it could be that the list is too exhaustive and a timeout is somewhere causing this to happen
• We were running our Terraform plan/apply on very small runners (0.5 vCPU, 1GiB RAM), after upgrading to larger sized runners (1 vCPU, 2GiB RAM) the error seems gone!

Expected Behavior

Consistent passing results or feedback about the issue at hand

Actual Behavior

Inconsistent results, most of the time failing, no feedback on the causing issue

Important Factoids

• ROSA (RHOS on AWS)
• GitLab CI

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[aristosvo]

I'm inclined to close this issue, as the underlying solution is available. It would be nice though to have a notification of the provider being memory deprived instead of just being killed, so I leave this open for @alexsomesan to judge whether we could somehow improve the error messaging

[BBBmau]

From Triage: Their may be a way to intercept the kill call regarding memory usage. Would require some investigation.

[aristosvo]

Hi @BBBmau 👋

Can you point me a bit in which direction to look? I have a bit of experience contributing to the Terraform providers for AWS and Azure, a pointer could help me do a bit of the work for you.