Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(opensearch): add support for latest amazon opensearch service 2.3 #22943

Merged
merged 13 commits into from
Jan 15, 2023

Conversation

agusti-t
Copy link
Contributor

fixes #22942
Description:
AWS Launched the support for Amazon OpenSearch Service 2.3 as per the announcement here: https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-opensearch-service-supports-opensearch-version-2-3/

AWS CDK does not support the latest version yet, and can not create an OpenSearch cluster with CDK using 2.3.


All Submissions:

Adding new Unconventional Dependencies:

  • This PR adds new unconventional dependencies following the process described here

New Features

  • Have you added the new feature to an integration test?
    • Did you use yarn integ to deploy the infrastructure and generate the snapshot (i.e. yarn integ without --dry-run)?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Nov 16, 2022

@github-actions github-actions bot added feature-request A feature should be added or improved. p2 beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK labels Nov 16, 2022
@aws-cdk-automation aws-cdk-automation requested a review from a team November 16, 2022 19:57
@agusti-t agusti-t force-pushed the OpenSearch_2_3 branch 3 times, most recently from 4346e92 to eecb475 Compare November 17, 2022 22:27
Copy link
Contributor

@TheRealAmazonKendra TheRealAmazonKendra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution!

The minimal changes in the snapshot of the integration test usually indicates that the tests weren't actually run. Please revert the change in the snapshot and run yarn integ --update-on-failed to validate this change and to produce the new snapshots. I also notice that the integ test isn't using the new IntegTest construct. Please update that.

@mergify mergify bot dismissed TheRealAmazonKendra’s stale review November 22, 2022 09:37

Pull request has been modified.

Copy link
Collaborator

@aws-cdk-automation aws-cdk-automation left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pull request linter has failed. See the aws-cdk-automation comment below for failure reasons. If you believe this pull request should receive an exemption, please comment and provide a justification.

@agusti-t agusti-t force-pushed the OpenSearch_2_3 branch 2 times, most recently from d2313a4 to 99d7141 Compare November 22, 2022 09:40
@agusti-t
Copy link
Contributor Author

agusti-t commented Nov 22, 2022

Thank you for your contribution!

The minimal changes in the snapshot of the integration test usually indicates that the tests weren't actually run. Please revert the change in the snapshot and run yarn integ --update-on-failed to validate this change and to produce the new snapshots. I also notice that the integ test isn't using the new IntegTest construct. Please update that.

Thanks @TheRealAmazonKendra for the review, I tried using the new IntegTest and run the tests as you said and I run into this problem:

➜  aws-cdk git:(OpenSearch_2_3) ✗ yarn integ-runner --directory packages/@aws-cdk/aws-appsync --update-on-failed
yarn run v1.22.19
$ /Users/agusti/src/aws-cdk/node_modules/.bin/integ-runner --directory packages/@aws-cdk/aws-appsync --update-on-failed

Verifying integration test snapshots...

  UNCHANGED  integ.appsync-none 0.915s
  UNCHANGED  integ.log-retention 1.066s
  UNCHANGED  integ.api-import 1.092s
  UNCHANGED  integ.appsync-lambda 1.082s
  CHANGED    integ.graphql-opensearch 1.214s
      appsync-opensearch exists in snapshot, but not in actual
  CHANGED    integ.graphql-opensearch 1.214s
      OpensSearch23Stack does not exist in snapshot, but does in actual
  UNCHANGED  integ.auth-apikey 1.92s
  UNCHANGED  integ.graphql-schema 1.919s
  UNCHANGED  integ.graphql-iam 1.948s
  UNCHANGED  integ.graphql-elasticsearch 1.972s
  UNCHANGED  integ.graphql 2.045s

Snapshot Results:

Tests:    1 failed, 10 total

Running integration tests for failed tests...

Running in parallel across regions: us-east-1, us-east-2, us-west-2
Running test /Users/agusti/src/aws-cdk/packages/@aws-cdk/aws-appsync/test/integ.graphql-opensearch.js in us-east-1
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!                                                                                  !!
!!  This software has not been tested with node v19.0.1.                            !!
!!  You may to encounter runtime issues, and should switch to a supported release.  !!
!!                                                                                  !!
!!  This software is currently running on node v19.0.1.                             !!
!!  As of the current release of this software, supported node releases are:        !!
!!  - ^18.0.0 (Planned end-of-life: 2025-04-30)                                     !!
!!  - ^16.3.0 (Planned end-of-life: 2023-09-11)                                     !!
!!  - ^14.6.0 (Planned end-of-life: 2023-04-30)                                     !!
!!                                                                                  !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

✨  Synthesis time: 0.3s

OpensSearch23Stack: building assets...

[0%] start: Building 7725c14cf18390409c0bad4e2d2037a44b300aac680ce62082d410f076e7f331:current_account-current_region
[100%] success: Built 7725c14cf18390409c0bad4e2d2037a44b300aac680ce62082d410f076e7f331:current_account-current_region

OpensSearch23Stack: assets built

lambda-api-latebound-deploymentstage/DefaultTest/DeployAssert: building assets...

[0%] start: Building 21fbb51d7b23f6a6c262b46a9caee79d744a3ac019fd45422d988b96d44b2a22:current_account-current_region
[100%] success: Built 21fbb51d7b23f6a6c262b46a9caee79d744a3ac019fd45422d988b96d44b2a22:current_account-current_region

lambda-api-latebound-deploymentstage/DefaultTest/DeployAssert: assets built

OpensSearch23Stack
OpensSearch23Stack: deploying...
[0%] start: Publishing 7725c14cf18390409c0bad4e2d2037a44b300aac680ce62082d410f076e7f331:current_account-current_region
[100%] success: Published 7725c14cf18390409c0bad4e2d2037a44b300aac680ce62082d410f076e7f331:current_account-current_region
OpensSearch23Stack: creating CloudFormation changeset...
OpensSearch23Stack | 0/10 | 12:38:57 AM | REVIEW_IN_PROGRESS   | AWS::CloudFormation::Stack     | OpensSearch23Stack User Initiated
OpensSearch23Stack | 0/10 | 12:39:04 AM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack     | OpensSearch23Stack User Initiated
OpensSearch23Stack | 0/10 | 12:39:09 AM | CREATE_IN_PROGRESS   | AWS::AppSync::GraphQLApi       | api (apiC8550315)
OpensSearch23Stack | 0/10 | 12:39:09 AM | CREATE_IN_PROGRESS   | AWS::IAM::User                 | User (User00B015A1)
OpensSearch23Stack | 0/10 | 12:39:10 AM | CREATE_IN_PROGRESS   | AWS::IAM::Role                 | api/ds/ServiceRole (apidsServiceRoleBDB08107)
OpensSearch23Stack | 0/10 | 12:39:10 AM | CREATE_IN_PROGRESS   | AWS::IAM::User                 | User (User00B015A1) Resource creation Initiated
OpensSearch23Stack | 0/10 | 12:39:10 AM | CREATE_FAILED        | AWS::IAM::Role                 | api/ds/ServiceRole (apidsServiceRoleBDB08107) Invalid principal in policy: "SERVICE":"appsync" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: d3941825-78dd-4440-9132-0254299af33f; Proxy: null)
OpensSearch23Stack | 0/10 | 12:39:13 AM | CREATE_IN_PROGRESS   | AWS::AppSync::GraphQLApi       | api (apiC8550315) Resource creation Initiated
OpensSearch23Stack | 1/10 | 12:39:13 AM | CREATE_COMPLETE      | AWS::AppSync::GraphQLApi       | api (apiC8550315)
OpensSearch23Stack | 1/10 | 12:39:16 AM | CREATE_IN_PROGRESS   | AWS::AppSync::GraphQLSchema    | api/Schema (apiSchema0EA92056)
OpensSearch23Stack | 1/10 | 12:39:18 AM | CREATE_IN_PROGRESS   | AWS::AppSync::GraphQLSchema    | api/Schema (apiSchema0EA92056) Resource creation Initiated
OpensSearch23Stack | 2/10 | 12:39:46 AM | CREATE_COMPLETE      | AWS::IAM::User                 | User (User00B015A1)
OpensSearch23Stack | 2/10 | 12:39:47 AM | CREATE_IN_PROGRESS   | AWS::OpenSearchService::Domain | Domain (Domain66AC69E0)
OpensSearch23Stack | 2/10 | 12:39:50 AM | CREATE_IN_PROGRESS   | AWS::OpenSearchService::Domain | Domain (Domain66AC69E0) Resource creation Initiated
OpensSearch23Stack | 3/10 | 12:40:19 AM | CREATE_COMPLETE      | AWS::AppSync::GraphQLSchema    | api/Schema (apiSchema0EA92056)
OpensSearch23Stack | 3/10 | 12:40:22 AM | CREATE_IN_PROGRESS   | AWS::AppSync::ApiKey           | api/DefaultApiKey (apiDefaultApiKey6AB8D7C4)
OpensSearch23Stack | 3/10 | 12:40:24 AM | CREATE_IN_PROGRESS   | AWS::AppSync::ApiKey           | api/DefaultApiKey (apiDefaultApiKey6AB8D7C4) Resource creation Initiated
OpensSearch23Stack | 4/10 | 12:40:24 AM | CREATE_COMPLETE      | AWS::AppSync::ApiKey           | api/DefaultApiKey (apiDefaultApiKey6AB8D7C4)
OpensSearch23Stack | 5/10 | 12:56:02 AM | CREATE_COMPLETE      | AWS::OpenSearchService::Domain | Domain (Domain66AC69E0)
OpensSearch23Stack | 5/10 | 12:56:03 AM | CREATE_FAILED        | AWS::CloudFormation::Stack     | OpensSearch23Stack The following resource(s) failed to create: [apidsServiceRoleBDB08107].

Failed resources:
OpensSearch23Stack | 12:39:10 AM | CREATE_FAILED        | AWS::IAM::Role                 | api/ds/ServiceRole (apidsServiceRoleBDB08107) Invalid principal in policy: "SERVICE":"appsync" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: d3941825-78dd-4440-9132-0254299af33f; Proxy: null)

 ❌  OpensSearch23Stack failed: Error: The stack named OpensSearch23Stack failed to deploy: CREATE_FAILED (The following resource(s) failed to create: [apidsServiceRoleBDB08107]. ): Invalid principal in policy: "SERVICE":"appsync" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: d3941825-78dd-4440-9132-0254299af33f; Proxy: null)
    at FullCloudFormationDeployment.monitorDeployment (/Users/agusti/src/aws-cdk/packages/aws-cdk/lib/api/deploy-stack.ts:505:13)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at deployStack (/Users/agusti/src/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:264:24)
    at /Users/agusti/src/aws-cdk/packages/aws-cdk/lib/deploy.ts:39:11
    at run (/Users/agusti/src/aws-cdk/node_modules/p-queue/dist/index.js:163:29)

 ❌ Deployment failed: Error: Stack Deployments Failed: Error: The stack named OpensSearch23Stack failed to deploy: CREATE_FAILED (The following resource(s) failed to create: [apidsServiceRoleBDB08107]. ): Invalid principal in policy: "SERVICE":"appsync" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: d3941825-78dd-4440-9132-0254299af33f; Proxy: null)
    at Object.exports.deployStacks (/Users/agusti/src/aws-cdk/packages/aws-cdk/lib/deploy.ts:61:11)
    at CdkToolkit.deploy (/Users/agusti/src/aws-cdk/packages/aws-cdk/lib/cdk-toolkit.ts:338:7)
    at initCommandLine (/Users/agusti/src/aws-cdk/packages/aws-cdk/lib/cli.ts:364:12)

Stack Deployments Failed: Error: The stack named OpensSearch23Stack failed to deploy: CREATE_FAILED (The following resource(s) failed to create: [apidsServiceRoleBDB08107]. ): Invalid principal in policy: "SERVICE":"appsync" (Service: AmazonIdentityManagement; Status Code: 400; Error Code: MalformedPolicyDocument; Request ID: d3941825-78dd-4440-9132-0254299af33f; Proxy: null)
  FAILED     integ.graphql-opensearch-lambda-api-latebound-deploymentstage/DefaultTest (undefined/us-east-1) 2389.226s
      Integration test failed: Error: Command exited with status 1

Test Results:

Tests:    1 failed, 1 total
Error: Some integration tests failed!
    at main (/Users/agusti/src/aws-cdk/packages/@aws-cdk/integ-runner/lib/cli.js:114:23)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Any idea what it could be? I tried looking it up without being able to find anything.

@TheRealAmazonKendra
Copy link
Contributor

@Mergifyio update

@mergify
Copy link
Contributor

mergify bot commented Nov 23, 2022

update

✅ Branch has been successfully updated

Copy link
Contributor

@TheRealAmazonKendra TheRealAmazonKendra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please see my comments inline.

We had a bug that was causing your test failures. I updated your branch to be up-to-date with main so the fix is in there now. You should now be able to run yarn integ --update-on-failed and get the correct output. Note that you may need to clean the generated files from the build because the test runner is expecting to see certain files that don't exist due to the update in stack name and using the new IntegTest construct.

Please let me know if you have any questions about this.

@aws-cdk-automation aws-cdk-automation dismissed their stale review November 24, 2022 19:17

✅ Updated pull request passes all PRLinter validations. Dissmissing previous PRLinter review.

@mergify mergify bot dismissed TheRealAmazonKendra’s stale review November 24, 2022 19:21

Pull request has been modified.

@agusti-t
Copy link
Contributor Author

agusti-t commented Nov 24, 2022

@TheRealAmazonKendra Trying again, it took me a while to be able to run the tests with yarn integ --update-on-failed, it kept complaining about not finding the integ cmd, TBH I don't know how/why it started working. I had to keep trying with yarn integ-runner --update-on-failed and that always failed. It could also be that I rebased main and got the bug (suspect this one) fix in, don't understand why I didn't get it earlier.

Anyway, it worked on my machine now and I committed everything, but I ask you to please check commits (1, 2) because I am not sure if these files should be committed. Of course the rest of the commits should be checked too.

@aws-cdk-automation
Copy link
Collaborator

This PR cannot be merged because it has conflicts. Please resolve them. The PR will be considered stale and closed if it remains in an unmergeable state.

@Naumel
Copy link
Contributor

Naumel commented Dec 8, 2022

@Mergifyio update

@mergify
Copy link
Contributor

mergify bot commented Dec 8, 2022

update

❌ Base branch update has failed

merge conflict between base and head
err-code: 21EBA

@agusti-t agusti-t force-pushed the OpenSearch_2_3 branch 5 times, most recently from 62a6fd5 to 7bfcec2 Compare January 14, 2023 15:42
@@ -183,8 +190,14 @@ test('grants kms permissions if needed', () => {

});

test('minimal example renders correctly', () => {
new Domain(stack, 'Domain', { version: defaultVersion });
each([
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic, thank you!

@mergify
Copy link
Contributor

mergify bot commented Jan 14, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildv2Project1C6BFA3F-wQm2hXv2jqQv
  • Commit ID: efe6e63
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit 0303d6f into aws:main Jan 15, 2023
@mergify
Copy link
Contributor

mergify bot commented Jan 15, 2023

Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

@agusti-t agusti-t deleted the OpenSearch_2_3 branch January 15, 2023 15:59
DerkSchooltink pushed a commit to DerkSchooltink/aws-cdk that referenced this pull request Jan 23, 2023
aws#22943)

fixes aws#22942 
Description:
AWS Launched the support for Amazon OpenSearch Service 2.3 as per the announcement here: https://aws.amazon.com/about-aws/whats-new/2022/11/amazon-opensearch-service-supports-opensearch-version-2-3/

AWS CDK does not support the latest version yet, and can not create an OpenSearch cluster with CDK using 2.3.

----

### All Submissions:

* [x] Have you followed the guidelines in our [Contributing guide?](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md)

### Adding new Unconventional Dependencies:

* [ ] This PR adds new unconventional dependencies following the process described [here](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md/#adding-new-unconventional-dependencies)

### New Features

* [ ] Have you added the new feature to an [integration test](https://github.com/aws/aws-cdk/blob/main/INTEGRATION_TESTS.md)?
	* [ ] Did you use `yarn integ` to deploy the infrastructure and generate the snapshot (i.e. `yarn integ` without `--dry-run`)?

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
beginning-contributor [Pilot] contributed between 0-2 PRs to the CDK feature-request A feature should be added or improved. p2
Projects
None yet
Development

Successfully merging this pull request may close these issues.

OpenSearch: Add support for OpenSearch 2.3
6 participants