Elasticsearch’s S3 Snapshot Repository: The Mystery of the Inaccessible S3 Bucket ==> Access Denied

[hackmoonlight]

Elasticsearch Version

7.17.12

Installed Plugins

repository-s3

Java Version

JAVA_RUNTIME_VERSION="20.0.2+9-78

OS Version

Linux elasticsearch-data-0 5.10.214-202.855.amzn2.x86_64 #1 SMP Tue Apr 9 06:57:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

I’m having a problem with Elasticsearch 7.17.12. I’m trying to set a ‘repository’ where Elasticsearch can store snapshots.
But, it seems like Elasticsearch can’t access the S3 bucket I’ve set up, even though everything looks set up correctly.

Steps to Reproduce

I’m encountering an issue when trying to create an S3 snapshot repository in Elasticsearch 7.17.12. I’m using the S3 repository plugin of the same version.

Here’s the command I’m using to create the repository:

curl -X PUT "http://localhost:9200/_snapshot/elasticsearch-data-backup" -H 'Content-Type: application/json' -u 'user:password' -k  -d' 
{
  "type": "s3",
  "settings": {
    "bucket": "elastic-storage-backup-dev",
    "endpoint": "s3.eu-west-3.amazonaws.com",
    "protocol": "https",
    "compress": true,
    "chunk_size": "100mb",
    "server_side_encryption": true 
    }
}'

output:
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node"}],"type":"repository_verification_exception","reason":"[elasticsearch-data-backup] path is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [tests-B77q-Ol-Qz6u5jNS1YR69A/master.dat] using a single upload","caused_by":{"type":"amazon_s3_exception","reason":"amazon_s3_exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: REZJX25X7E2TJWCZ; S3 Extended Request ID: 4e``

I’ve set up an IAM role with the necessary permissions and attached it to the Elasticsearch service. The IAM role has the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.eu-west-3.amazonaws.com/id/88D51DF3F1A6FFD0CC555D88736D4F31:sub": [
                        "system:serviceaccount:default:elasticsearch-data",
                        "system:serviceaccount:default:elasticsearch-master"
                    ]
                }
            }
        }
    ]
}

The S3 bucket policy is as follows:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<account-id>role/elastic-dev"
            },
            "Action": [
                "s3:GetObject",
                "s3:GetBucketLocation",
                "s3:ListBucket",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucketMultipartUploads",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts",
                "s3:GetObjectVersion"
            ],
            "Resource": [
                "arn:aws:s3:::elastic-storage-backup-dev",
                "arn:aws:s3:::elastic-storage-backup-dev/*"
            ]
        }
    ]
}

The Elasticsearch service and the S3 bucket are both in the eu-west-3 region. Public access is blocked for the S3 bucket.

Despite this, I’m still encountering the “Access Denied” error when trying to create the S3 snapshot repository. Any help would be greatly appreciated.

Logs (if relevant)

No response

[DaveCTurner]

Thanks very much for your interest in Elasticsearch.

This appears to be a user question, and we'd like to direct these kinds of things to the Elasticsearch forum. If you can stop by there, we'd appreciate it. This allows us to use GitHub for verified bug reports, feature requests, and pull requests.

Specifically, I think the problem you are facing is most likely a consequence of how you are configuring your environment in AWS and won't require any code changes to Elasticsearch to address. This isn't the right place to investigate environmental config issues.

There's an active community in the forum that should be able to help get an answer to your question. As such, I hope you don't mind that I close this.