[Bug]: Error: Plugin did not respond fatal error: out of memory during ReadDataSource

[albadrnasution]

Terraform Core Version

1.3.2,1.4.6

AWS Provider Version

4.67.0_x5

Affected Resource(s)

data.aws_instance

Expected Behavior

AWS has 8000 instances of EC2 with the filter. Data should have been read.

Actual Behavior

Terraform failed with Error: Plugin did not respond

Relevant Error/Panic Output Snippet

data.aws_instances.ba_device: Reading...
data.aws_instances.ba_device: Still reading... [10s elapsed]
data.aws_instances.ba_device: Still reading... [20s elapsed]
data.aws_instances.ba_device: Still reading... [30s elapsed]
╷
│ Error: Plugin did not respond
│
│   with data.aws_instances.ba_device,
│   on resource.tf line 1, in data "aws_instances" "ba_device":
│    1: data "aws_instances" "ba_device" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).ReadDataSource call. The plugin logs may contain more details.
╵

Stack trace from the terraform-provider-aws_v4.67.0_x5.exe plugin:

fatal error: out of memory

goroutine 380 [running]:
runtime.throw({0xc3a14bd, 0xd})
        runtime/panic.go:1047 +0x4d fp=0x2b291288 sp=0x2b291274 pc=0x7289ad
runtime.(*mcache).allocLarge(0x166f09c8, 0x757e000, 0x1)
        runtime/mcache.go:235 +0x208 fp=0x2b2912b0 sp=0x2b291288 pc=0x704928
runtime.mallocgc(0x757e000, 0x0, 0x0)
        runtime/malloc.go:1029 +0x446 fp=0x2b2912f8 sp=0x2b2912b0 pc=0x6fbaf6
runtime.growslice(0xa5d9700, {0x63eb0000, 0x5dfe000, 0x5dfe000}, 0x5dfe001)
        runtime/slice.go:284 +0x3d7 fp=0x2b291324 sp=0x2b2912f8 pc=0x73d017
io.ReadAll({0xd78d9f0, 0x2a1d6570})
        io/io.go:659 +0x8b fp=0x2b291364 sp=0x2b291324 pc=0x7c0ccb
github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2.decomposeResponseBody({0xd78d9f0, 0x2a1d6570})
        github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.26/logger.go:187 +0x38 fp=0x2b2913c8 sp=0x2b291364 pc=0x7ee36b8
github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2.decomposeHTTPResponse(0x2a1da5f0, {0xd78d9f0, 0x2a1d6570}, 0x6f247f86c)
        github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.26/logger.go:172 +0x30e fp=0x2b29150c sp=0x2b2913c8 pc=0x7ee345e
github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2.logResponse.func1(0x2a604000)
        github.com/hashicorp/aws-sdk-go-base/v2/awsv1shim/v2@v2.0.0-beta.26/logger.go:133 +0x133 fp=0x2b291584 sp=0x2b29150c pc=0x7ee3003
github.com/aws/aws-sdk-go/aws/request.(*HandlerList).Run(0x2a604138, 0x2a604000)
        github.com/aws/aws-sdk-go@v1.44.261/aws/request/handlers.go:267 +0x66 fp=0x2b2915cc sp=0x2b291584 pc=0xf6be96
github.com/aws/aws-sdk-go/aws/request.(*Request).sendRequest(0x2a604000)
        github.com/aws/aws-sdk-go@v1.44.261/aws/request/request.go:620 +0x10d fp=0x2b29161c sp=0x2b2915cc pc=0xf6f1fd
github.com/aws/aws-sdk-go/aws/request.(*Request).Send(0x2a604000)
        github.com/aws/aws-sdk-go@v1.44.261/aws/request/request.go:555 +0x13a fp=0x2b29166c sp=0x2b29161c pc=0xf6eaaa
github.com/aws/aws-sdk-go/aws/request.(*Pagination).Next(0x2b2916b0)
        github.com/aws/aws-sdk-go@v1.44.261/aws/request/request_pagination.go:107 +0xb6 fp=0x2b29169c sp=0x2b29166c pc=0xf70056
github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPagesWithContext(0x29f6f4d8, {0xd7d3080, 0x2a2a2288}, 0x2b78f440, 0x2b291750, {0x0, 0x0, 0x0})
        github.com/aws/aws-sdk-go@v1.44.261/service/ec2/api.go:23097 +0xf8 fp=0x2b291714 sp=0x2b29169c pc=0x4a31b28
github.com/hashicorp/terraform-provider-aws/internal/service/ec2.FindInstances({0xd7d3080, 0x2a2a2288}, 0x29f6f4d8, 0x2b78f440)
        github.com/hashicorp/terraform-provider-aws/internal/service/ec2/find.go:1033 +0x97 fp=0x2b291768 sp=0x2b291714 pc=0x82795a7
github.com/hashicorp/terraform-provider-aws/internal/service/ec2.dataSourceInstancesRead({0xd7d3080, 0x2a2a2288}, 0x2b6ffef0, {0xc3363a0, 0x27a52c00})
        github.com/hashicorp/terraform-provider-aws/internal/service/ec2/ec2_instances_data_source.go:92 +0x6da fp=0x2b291868 sp=0x2b291768 pc=0x822faca
github.com/hashicorp/terraform-provider-aws/internal/provider.interceptedHandler[...].func1(0x2b6ffef0, {0xc3363a0, 0x27a52c00})
        github.com/hashicorp/terraform-provider-aws/internal/provider/intercept.go:95 +0x182 fp=0x2b291908 sp=0x2b291868 pc=0x99092f2
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).read(0x274fb5e0, {0xd7d3080, 0x2b6c3d70}, 0x2b6ffef0, {0xc3363a0, 0x27a52c00})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:719 +0x80 fp=0x2b29193c sp=0x2b291908 pc=0xe5b7c0
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ReadDataApply(0x274fb5e0, {0xd7d3080, 0x2b6c3d70}, 0x2b6ffe50, {0xc3363a0, 0x27a52c00})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/resource.go:943 +0x12f fp=0x2b29197c sp=0x2b29193c pc=0xe5d2ef
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadDataSource(0x29b7fd80, {0xd7d3080, 0x2b6c3d10}, 0x2926d7f0)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.26.1/helper/schema/grpc_provider.go:1195 +0x3f5 fp=0x2b2919fc sp=0x2b29197c pc=0xe56245
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.muxServer.ReadDataSource({0x29c53100, 0x29c53140, {0x2b1da8f0, 0x2, 0x2}, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}, ...}, ...)
        github.com/hashicorp/terraform-plugin-mux@v0.10.0/tf5muxserver/mux_server_ReadDataSource.go:27 +0x151 fp=0x2b291a40 sp=0x2b2919fc pc=0xd3cc11
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.(*muxServer).ReadDataSource(0x289eafc0, {0xd7d3080, 0x2b6c3c20}, 0x2926d7f0)
        <autogenerated>:1 +0x6d fp=0x2b291b0c sp=0x2b291a40 pc=0xd3e02d
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadDataSource(0x2ae839f0, {0xd7d3080, 0x2b6c3968}, 0x2b78eed0)
        github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/tf5server/server.go:658 +0x436 fp=0x2b291cd4 sp=0x2b291b0c pc=0xc632d6
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler({0xc1bfd40, 0x2ae839f0}, {0xd7d3080, 0x2b6c3968}, 0x2a203300, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:455 +0x18a fp=0x2b291d08 sp=0x2b291cd4 pc=0xc4d06a
google.golang.org/grpc.(*Server).processUnaryRPC(0x2ae738c0, {0xd7d9070, 0x2b332000}, 0x2a23af00, 0x2b1dd590, 0x13e9b7f8, 0x0)
        google.golang.org/grpc@v1.54.0/server.go:1345 +0xed7 fp=0x2b291f2c sp=0x2b291d08 pc=0xbd20d7
google.golang.org/grpc.(*Server).handleStream(0x2ae738c0, {0xd7d9070, 0x2b332000}, 0x2a23af00, 0x0)
        google.golang.org/grpc@v1.54.0/server.go:1722 +0xa17 fp=0x2b291fb8 sp=0x2b291f2c pc=0xbd68f7
google.golang.org/grpc.(*Server).serveStreams.func1.2()
        google.golang.org/grpc@v1.54.0/server.go:966 +0xb1 fp=0x2b291ff0 sp=0x2b291fb8 pc=0xbcfaf1
runtime.goexit()
        runtime/asm_386.s:1326 +0x1 fp=0x2b291ff4 sp=0x2b291ff0 pc=0x755311
created by google.golang.org/grpc.(*Server).serveStreams.func1
        google.golang.org/grpc@v1.54.0/server.go:964 +0x25f

goroutine 1 [select]:
runtime.gopark(0xc5ff38c, 0x0, 0x9, 0x18, 0x1)
        runtime/proc.go:363 +0xff fp=0x274b1acc sp=0x274b1ab8 pc=0x72b80f
runtime.selectgo(0x274b1d38, 0x2766fc5c, 0x0, 0x0, 0x2, 0x1)
        runtime/select.go:328 +0xbe8 fp=0x274b1bc0 sp=0x274b1acc pc=0x73a428
github.com/hashicorp/go-plugin.Serve(0x2b1d6ba0)
        github.com/hashicorp/go-plugin@v1.4.9/server.go:474 +0x1530 fp=0x274b1e4c sp=0x274b1bc0 pc=0xc01af0
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.Serve({0xc4b80b9, 0x23}, 0x289eab40, {0x0, 0x0, 0x0})
        github.com/hashicorp/terraform-plugin-go@v0.15.0/tfprotov5/tf5server/server.go:312 +0xc3f fp=0x274b1f84 sp=0x274b1e4c pc=0xc6018f
main.main()
        github.com/hashicorp/terraform-provider-aws/main.go:32 +0x1f9 fp=0x274b1fc4 sp=0x274b1f84 pc=0x990a199
runtime.main()
        runtime/proc.go:250 +0x231 fp=0x274b1ff0 sp=0x274b1fc4 pc=0x72b451
runtime.goexit()
        runtime/asm_386.s:1326 +0x1 fp=0x274b1ff4 sp=0x274b1ff0 pc=0x755311

goroutine 2 [force gc (idle)]:
runtime.gopark(0xc5ff360, 0x13ea5f88, 0x11, 0x14, 0x1)
        runtime/proc.go:363 +0xff fp=0x274abfdc sp=0x274abfc8 pc=0x72b80f
runtime.goparkunlock(...)
        runtime/proc.go:369
runtime.forcegchelper()
        runtime/proc.go:302 +0xc3 fp=0x274abff0 sp=0x274abfdc pc=0x72b693
runtime.goexit()
        runtime/asm_386.s:1326 +0x1 fp=0x274abff4 sp=0x274abff0 pc=0x755311
created by runtime.init.5
        runtime/proc.go:290 +0x23

goroutine 3 [runnable]:
runtime.Gosched(...)
        runtime/proc.go:318
runtime.bgsweep(0x2747a6c0)
        runtime/mgcsweep.go:283 +0x11b fp=0x274acfe8 sp=0x274acfd4 pc=0x7152eb
runtime.gcenable.func1()
        runtime/mgc.go:178 +0x27 fp=0x274acff0 sp=0x274acfe8 pc=0x706ec7
runtime.goexit()
        runtime/asm_386.s:1326 +0x1 fp=0x274acff4 sp=0x274acff0 pc=0x755311
created by runtime.gcenable
        runtime/mgc.go:178 +0x7c

goroutine 4 [GC scavenge wait]:

Error: The terraform-provider-aws_v4.67.0_x5.exe plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Configuration Files

data "aws_instances" "ba_device" {
  filter {
    name   = "instance-state-name"
    values = ["running"]
  }
  instance_tags = {
    "Prefix"    = var.prefix
    "Component" = "BotAgentDevice"
  }
}

locals {
  instance_ids = data.aws_instances.ba_device.ids
  passwords    = data.aws_instance.ba_devices.*.tags.Password
  host_ips     = data.aws_instance.ba_devices.*.public_ip
  time         = formatdate("YYYYMMDD-hhmm", timestamp())
  env_name = var.prefix
}

data "aws_instance" "ba_devices" {
  count       = length(local.instance_ids)
  instance_id = local.instance_ids[count.index]
}

variable "prefix" {
  default = "dtr-29-cloud-toptier"
}

output "ids" {
  value = data.aws_instances.ba_device.ids
}

Steps to Reproduce

terraform plan
terraform apply -auto-approve

Debug Output

No response

Panic Output

No response

Important Factoids

The aws_instance count to be read are quite big: 8000 instances.

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @albadrnasution 👋 Thank you for taking the time to raise this! I imagine that reading 8000 instances would use quite a bit of memory -- are you able to monitor the system's memory to verify whether this might be a case where the memory is actually being exhausted by the run?

[albadrnasution]

Hi @justinretzolk Thanks for the reply. I tried 2 times in different system. One inside our terraform container (kubernetes) and one on my Windows laptop. Both has ample memory. No discernable memory lacking during the run. The container has 2 GB memory request/limit and my laptop has 32GB RAM (some giga available during run). In windows, I can see that terraform-provider-aws_v4.67.0_x5.exe consume 694 MB before it failed with OOM.

For comparison, I tried aws cli describe-instance command and it works as expected, even with 8000 ec2 instances.

[justinretzolk]

Hey @albadrnasution 👋 Thank you for getting back to me with that information! I'm going to get this marked as a bug so that the team can take a look into this when possible.

Interestingly, I happened to see another issue raised that I would otherwise consider entirely unrelated, save for the fact that both of you are running 4.67.0 on Windows, and are both getting the fatal: out of memory error when ample system memory is seemingly available. Linking these in case they're somehow related #31571.

[github-actions]

This functionality has been released in v5.1.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.