Warning: Response body is not recognized as UTF-8 even if response_body is not used

[sashee]

Terraform CLI and Provider Versions

Terraform v1.5.1
on linux_amd64

• provider registry.terraform.io/hashicorp/aws v5.5.0
• provider registry.terraform.io/hashicorp/http v3.4.0
• provider registry.terraform.io/hashicorp/local v2.4.0
• provider registry.terraform.io/hashicorp/random v3.5.1

Terraform Configuration

provider "aws" {
}

resource "random_id" "id" {
  byte_length = 8
}

data "http" "image" {
	url = "https://unsplash.com/photos/F3rDBnQQbQU/download?force=true&w=1300"
}

resource "local_file" "image" {
  content_base64  = data.http.image.response_body_base64
  filename = "/tmp/img-${random_id.id.hex}"
}

resource "aws_s3_object" "images" {
  key    = "testimage"
	source = local_file.image.filename
  bucket = aws_s3_bucket.bucket.bucket
  etag   = local_file.image.content_md5
}

resource "aws_s3_bucket" "bucket" {
  force_destroy = "true"
}

Expected Behavior

Apply should not report any warnings.

Actual Behavior

During plan I see a warning:

Plan: 4 to add, 0 to change, 0 to destroy.
╷
│ Warning: Response body is not recognized as UTF-8
│
│   with data.http.image,
│   on main.tf line 8, in data "http" "image":
│    8: data "http" "image" {
│
│ Terraform may not properly handle the response_body if the contents are binary.
╵

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

Since I don't use the response_body there is no problem with binary data.

Steps to Reproduce

1. terraform plan

How much impact is this issue causing?

Low

Logs

https://gist.github.com/sashee/ee2392c311a64ec0a1f5789b319528f0

Additional Information

Getting the file using the content_base64 works fine, so the warning is only a small issue here.

Code of Conduct

• I agree to follow this project's Code of Conduct

[bendbennett]

Hi @sashee 👋

Owing to the issues that can arise when binary data is used in Terraform configuration, the changes in Base64 encode response body (released in v3.4.0) added a warning to flag when the response_body is not recognised as UTF-8. Unfortunately, Terraform has no way to signal to the http provider whether the response_body is used in any other Terraform configuration so it's not possible to only raise a warning when the response_body is used.

The only current options we have are:

• Removing the warning diagnostic and relying on the documentation to denote why the response_body attribute value may not be usable by Terraform.
• Demoting the warning diagnostic to a log message, which is easily missed, and this still has the same issue of relying on the documentation to denote why the response_body attribute value may not be usable by Terraform.
• Adding a configuration attribute to disable the warning (essentially opt into understanding the risks).

Another possibility might be to raise a feature request on Terraform core requesting that a provider be given additional information about whether an attribute value is actually being used in this type of situation. There is a similar issue regarding computed attributes that are deprecated.

[sashee]

Hi @bendbennett ,

Thanks for the thoughtful response!

My initial thought was to just add an attribute (your 3rd point) but instead of controlling the warning it could control the parsing. For example, something like parse_response_as_text or similar that would explicitly tell the provider not to try to parse the response as text. In that case, the response_body could be null as well, to avoid the case where it is used with binary data.

[bendbennett]

Hi @sashee,

Thank you for the suggestion. We will consider your proposal alongside the other options outlined above in the context of the amount of community interest in this issue.

[yirui-openai]

Hi @bendbennett , I agree with @sashee where allowing caller to explicitly signal the resource (i.e. the 3rd point) on the type of data and suppress the warning would be nice. We are relying on it to download a simple cert (which is not a text file), and all places are already using response_body_base64, having this warning could be confusing.