update AWS SDK for ECS Task IAM support in discovery-ec2

[mohit]

Updating the AWS SDK to the latest released version. In particular this support ECS IAM Task Roles which are very useful when running Elasticsearch in a docker container on ECS.

Fixes #23039 for discovery-ec2. Using this on ECS and tested it locally using docker.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[jasontedor]

There is more to updating dependencies than bumping the version. You need to also update SHAs and maybe grant permissions to the JAR over the old version. It could be the new SDK needs more permissions than the current, and those might be permissions we would not even grant. So I would ask you: please at least run gradle :plugins:discovery-ec2:check and update this pull request accordingly.

[dadoonet]

I think we should update both discovery-ec2 and repository-s3 projects at the same time.

Note that last time I tried to upgrade AWS SDK, I got a SM issue (See #19594). May be it's not the case anymore but @mohit could you try to run your own version of the plugin and see if it does not break anything?

Then, does this upgrade automagically add support for ECS Task? Or is there any other code change need?

Thanks!

[mohit]

Thanks for the instructions! I found various Mocks for AwsEc2Service that broke and am in the process of adding the fixes. As for the runtime, I replaced the jars in my Docker container for discovery-ec2 and it has been running the cluster on ECS successfully over the last couple of weeks with add/remove nodes.

No code changes (apart from tests) are required for discovery-ec2 since it uses the DefaultCredentialsChain but repository-s3 requires some changes.

I'll work on pushing the changes for repository-s3 after I'm done with discovery-ec2.

My current experiments with repository-s3 have involved using the DefaultAWSCredentialsProviderChain instead of the PrivilegedInstanceProfileCredentialsProvider in the plugin: https://github.com/elastic/elasticsearch/blob/master/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/InternalAwsS3Service.java#L135

This comment suggests not using the DefaulAWSCredentialsProviderChain for security reasons but in the code I see env vars (or StaticCredentials) are still supported.

In either case, thanks for the help and I'll add the changes in the next 24 hours or so (working with Java after a long time and it's taking some time to bootstrap the JDK/Eclipse environment).

[mohit]

@dadoonet Apologies but I've updated the AmazonEC2Mock and license SHAs but currently getting the following error when running gradle plugins:discovery-ec2:check and not sure what it is referring to.

* What went wrong:
Execution failed for task ':plugins:discovery-ec2:thirdPartyAudit'.
> CLASSES ARE MISSING! [com/amazonaws/jmespath/JmesPathEvaluationVisitor.class, com/amazonaws/jmespath/JmesPathExpression.class, com/amazonaws/jmespath/JmesPathField.class, com/amazonaws/jmespath/JmesPathFlatten.class, com/amazonaws/jmespath/JmesPathIdentity.class, com/amazonaws/jmespath/JmesPathLengthFunction.class, com/amazonaws/jmespath/JmesPathLiteral.class, com/amazonaws/jmespath/JmesPathProjection.class, com/amazonaws/jmespath/JmesPathSubExpression.class, com/amazonaws/jmespath/ObjectMapperSingleton.class, com/amazonaws/jmespath/OpGreaterThan.class, software/amazon/ion/IonReader.class, software/amazon/ion/IonSystem.class, software/amazon/ion/IonType.class, software/amazon/ion/IonWriter.class, software/amazon/ion/Timestamp.class, software/amazon/ion/system/IonBinaryWriterBuilder.class, software/amazon/ion/system/IonSystemBuilder.class, software/amazon/ion/system/IonTextWriterBuilder.class, software/amazon/ion/system/IonWriterBuilder.class]

[jasontedor]

@mohit Those are classes referenced by dependencies that are not in any of the dependencies on the classpath. You have to figure out if those classes are needed for the functionality we support, or not. If they are needed, you need to add the required dependency. If they are not, they need to be excluded for the audit (in the build.gradle) explaining why. We do not like adding new dependencies so the case must be strong that a new one is needed here.

[jasontedor]

Also, I think it will be too much to add upgrading repository-s3 to this PR, please leave it out.

[mohit]

I don't think the classes are required on account of me already having used the InstanceProfile and ContainerProfile features of the AWS SDK library. Following up on other explanations in build.gradle to see if I can get more rigorous about these exclusions.

[mohit]

With the above commits gradle plugins:discovery-ec2:check is passing. Going to make a different PR for repository-s3 which seems like a little bit more work.

[dadoonet]

Also, I think it will be too much to add upgrading repository-s3 to this PR, please leave it out.

I agree that it's not mandatory. We need to open an issue (or better another PR) to track it IMO.

[dadoonet]

@mohit Just checking: did you try to run your version of the plugin on EC2?
Does it support ECS Task IAM? If so, is there anything to add in the documentation?

[mohit]

@dadoonet I have tested by adding the updated AWS dependencies to the docker.elastic.co/elasticsearch/elasticsearch:5.5.2 docker image and discovery-ec2 does inherit ECS Task IAM Role assigned.

Unfortunately I've not been able to get a working elasticsearch:7.0.0-alpha1 container built to install this plugin. The installation of my discovery-ec2 build on the elasticsearch:5.5.2 docker container is complaining about a version mismatch. Exception in thread "main" java.lang.IllegalStateException: /usr/share/elasticsearch/plugins/.installing-5410796145994513969/discovery-ec2-7.0.0-alpha1-SNAPSHOT.jar requires Elasticsearch 7.0.0-alpha1, your system: 5.5.2

Any advice on how to proceed will be welcome. Thanks again on all your help!

[jasontedor]

@mohit I would recommend cherry-picking your changes to the plugin to the v6.0.0-beta2 tag, building that from source, and using the 6.0.0-beta2 version of our Docker container for testing.

[mohit]

Okie. So I built a container using 6.0.0-beta1 but it seems that cloud.aws.region has been deprecated. This setting however, seems to be required for discovery-ec2 to work properly in the AWS region I'm testing us-west-1 when using 5.5.

I'm not a 100% sure why the zen discovery is failing to find other nodes in 6.0.0 but it seems to show the same behavior as 5.5 when cloud.aws.region is not set. I can dig into alternatives and debug level logging but would love to know more about the alternative for setting AWS SDK region in 6.0.0-beta

https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2-usage.html#discovery-ec2-usage-region

[mohit]

okay found https://www.elastic.co/guide/en/elasticsearch/plugins/6.0/_settings.html, will test by setting endpoint in 6.0.0-beta2. I don't think the automatic setting for this is working correctly though.

[mohit]

Okay was able to build a docker container using 6.0.0-beta2 and run it on ECS using the commits cherry-picked from this branch. The cluster zen discovery works and checked CloudTrail logs to ensure that the IAM Task Role was being used for the requests to AWS.

    "userIdentity": {
        "type": "AssumedRole",
        "principalId": "AAAAAAAA-5d1ff6c2692d",
        "arn": "arn:aws:sts::000000000:assumed-role/elasticsearch-dev/-0000-5d1ff6c2692d",
        "accountId": "000000000",
        "accessKeyId": "000000000AAA",
        "sessionContext": {
            "attributes": {
                "mfaAuthenticated": "false",
                "creationDate": "2017-09-07T00:12:50Z"
            },
            "sessionIssuer": {
                "type": "Role",
                "principalId": "000000000AAAA",
                "arn": "arn:aws:iam::000000000:role/elasticsearch-dev",
                "accountId": "000000000",
                "userName": "elasticsearch-dev"
            }
        }
    },
    "eventTime": "2017-09-07T00:15:14Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "DescribeInstances",
    "awsRegion": "us-west-1",
    "sourceIPAddress": "00.000.000.000",
    "userAgent": "aws-sdk-java/1.11.187 Linux/4.4.41-35.53.amzn1.x86_64 OpenJDK_64-Bit_Server_VM/25.141-b16/1.8.0_141",
    "requestParameters": {
        "instancesSet": {},
        "filterSet": {
            "items": [
                {
                    "name": "instance-state-name",
                    "valueSet": {
                        "items": [
                            {
                                "value": "running"
                            },
                            {
                                "value": "pending"
                            }
                        ]
                    }
                }
            ]
        }
    }

[mohit]

I don't think there is a need to add anything to the documentation but like I mentioned above setting discovery.ec2.endpoint was required and the automatic region detection is not working.

[mohit]

@dadoonet any other requirements here?

[dadoonet]

LGTM
I can hopefully merge it next week unless @jasontedor wants to do it?

[dadoonet]

Thanks @mohit. I pushed it in master (7.0.0) and 6.x (6.1.0).