aws_sqs source: auth.credentials_file option does not work

[hhromic]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Vector Version

vector 0.19.1 (x86_64-unknown-linux-gnu 3cf70cf 2022-01-25)

Vector Configuration File

Consider the following Vector configuration file:

sources:
  my-source:
    type: aws_sqs
    region: YOUR_REGION
    queue_url: YOUR_QUEUE_URL
    auth:
      credentials_file: /path/to/credentials

sinks:
  console:
    type: console
    inputs:
      - my-source
    encoding:
      codec: json

Expected Behavior

According to the documentation for the auth.credentials_file option:

The path to AWS credentials file. Used for AWS authentication when communicating with AWS services.

If there is a file in /path/to/credentials with this content:

[default]
aws_access_key_id = YOUR_KEY_ID
aws_secret_access_key = YOUR_ACCESS_KEY

Then Vector should be able to pick it for authenticating with the AWS services.

Actual Behavior

The AWS SDK seems to not like to be passed a credentials file (maybe used to but not anymore?) from Vector:

2022-01-28T16:45:54.648223Z  WARN vector::aws::aws_sdk::auth: Overriding the credentials file is not supported. `~/.aws/config` and `~/.aws/credentials` will be used instead of "/path/to/credentials"

The result is that the credentials file specified in auth.credentials_file is completely ignored and then this error appears:

2022-01-28T16:50:08.708706Z ERROR vector::sources::aws_sqs::source: SQS receive message error: ConstructionFailure(MissingCredentials)

However, the corresponding environment variable for setting the credentials file DOES work correctly, but you have to NOT use auth.credentials_file in the Vector configuration itself.

AWS_SHARED_CREDENTIALS_FILE=/path/to/credentials vector -c test.yaml
2022-01-28T16:54:02.635898Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info"
2022-01-28T16:54:02.636161Z  INFO vector::app: Loading configs. paths=["test.yaml"]
2022-01-28T16:54:03.657047Z  INFO vector::topology::running: Running healthchecks.
2022-01-28T16:54:03.657488Z  INFO vector::topology::running: Starting source. key=my-source
2022-01-28T16:54:03.658073Z  INFO vector::topology::running: Starting sink. key=console
2022-01-28T16:54:03.658676Z  INFO vector: Vector has started. debug="false" version="0.19.1" arch="x86_64" build_id="3cf70cf 2022-01-25"
2022-01-28T16:54:03.659031Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2022-01-28T16:54:03.666742Z  INFO vector::topology::builder: Healthcheck: Passed.
(...messages from the configured SQS queue appear here...)

If the AWS SDK does not allow for a credentials file to be passed anymore, maybe is best to remove this option from Vector altogether to avoid frustration with users. Or maybe there is another way nowadays in the SDK to pass a credentials file?

[jszwedko]

Hi @hhromic !

Thanks for reporting!

This is an artifact of the aws_sqs source using the new AWS SDK, which doesn't support passing the files to read as an option. I agree that we should prohibit setting the option at all on the aws_sqs source. I think it was left because the configuration struct is shared with the other AWS components that use the old AWS SDK (rusoto) which did support this functionality.

As a work-around, the new AWS SDK does support configuring the profile and credentials file location via environment variables: AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE.

I'll check if they plan to add support for configuring the file to aws-sdk-rust. If they don't, we can plan to drop support for configuring it and Vector just document the existence of environment variables.

[jszwedko]

Looks like they may plan to be adding support: awslabs/aws-sdk-rust#237

[hhromic]

As a work-around, the new AWS SDK does support configuring the profile and credentials file location via environment variables: AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE.

Yup, as stated in the original issue, we indeed discovered that the environment variables do work properly as they are picked up by the AWS SDK directly. 👍

Looks like they may plan to be adding support: awslabs/aws-sdk-rust#237

If they are going to bring support for this in the future, then I guess the option can stay in Vector but I would suggest then to document clearly that this option currently does not work due to using a new AWS SDK and to use env variables in the meantime. This way users familiar with related sources, i.e. aws_s3 would know what to expect.

Thanks for checking the issue anyway!