Empty or missing Vault Agent token files are not re-read

[twispt]

If the specified vault_agent_token_file is empty or missing, it will not be re-read after being populated. This issue was introduced in version 0.29.3.

We are able to workaround this issue by pre-populating the token file with a placeholder value.

Consul Template version

Reproducible with 0.39.1 and presumably any version after 0.29.3. Version 0.29.2 (and earlier) will re-read the token file successfully.

Configuration

This config has generous retry attempts to make it easier to test. Versions less than 0.29.3 will successfully re-read the token file during a retry attempt and successfully render templates.

vault {
  address = "http://127.0.0.1:8200"
  vault_agent_token_file = "./token"
  retry {
    enabled = true
    attempts = 20
    backoff = "3s"
    max_backoff = "3s"
  }
}

template {
  source      = "one.txt.ctmpl"
  destination = "one.txt"
  error_on_missing_key = true
  backup = false
}

{{- with secret "secret/one" }}{{ .Data.data.my_key }}{{- end }}

{
  "my_key": "my_value"
}

Command

/consul-template-0.29.3 -log-level=debug -config config.hcl

Debug output

Version 0.39.1 Vault token file was empty at the start and was populated at about retry 6

Version 0.39.1 with placeholder token value This time the token file contained a placeholder value at the start and succeeds after a valid token is set

Version 0.29.2 Vault token file was empty at the start

Expected behavior

When a token file is created and populated during the Vault retry loop it should be re-read by the file watcher and used in subsequent requests.

Actual behavior

All Vault retries are exhausted despite the token file containing a valid token.

Steps to reproduce

1. Ensure the token file referenced in config is empty or missing
2. Start consul-template using the command above. Note failure and retry log messages
3. While retries are on-going populate the file with a valid token
4. 0.29.3 and up will exhaust all retries and fail, lesser versions will succeed on the next retry

References

• https://github.com/hashicorp/consul-template/blob/main/CHANGELOG.md#v0293-sept-30-2022