Support Eventbridge Pipes parameters #31607

jordanfinners · 2023-05-26T21:18:33Z

Description

Add support for Eventbridge Enrichment Parameters
Add support for Eventbridge Target Parameters
Add support for Eventbridge Source Parameters
Due to the amount of parameters add unit tests to ensure the mapping is correct to the right struct value
Update docs to support parameters

Relations

Closes #28153.
Closes #31330.
Closes #32018.

Output from Acceptance Testing

I've got most of the tests working however I'm having issues with, I don't have much time anymore to dedicate to debugging these, so need some help with these please @ewbankkit:

TestAccPipesPipe_source_sqs_target_batch_job
TestAccPipesPipe_source_sqs_target_ecs_task
TestAccPipesPipe_source_managed_streaming_kafka_target_sqs
TestAccPipesPipe_source_self_managed_streaming_kafka_target_sqs

Overall:

TestAccPipesPipe_basic - PASS
TestAccPipesPipe_disappears - PASS
TestAccPipesPipe_description - PASS
TestAccPipesPipe_desiredState - PASS
TestAccPipesPipe_enrichment - PASS
TestAccPipesPipe_sourceParameters_filterCriteria - PASS
TestAccPipesPipe_nameGenerated - PASS
TestAccPipesPipe_namePrefix - PASS
TestAccPipesPipe_roleARN - PASS
TestAccPipesPipe_tags - PASS
TestAccPipesPipe_source_sqs_target_sqs - PASS
TestAccPipesPipe_source_kinesis_target_kinesis - PASS
TestAccPipesPipe_source_dynamo_target_cloudwatch_logs - PASS
TestAccPipesPipe_source_active_mq_target_sqs - PASS
TestAccPipesPipe_source_rabbit_mq_target_sqs - PASS
TestAccPipesPipe_source_managed_streaming_kafka_target_sqs
TestAccPipesPipe_source_self_managed_streaming_kafka_target_sqs
TestAccPipesPipe_source_sqs_target_batch_job
TestAccPipesPipe_source_sqs_target_event_bridge_event_bus - PASS
TestAccPipesPipe_source_sqs_target_http - PASS
TestAccPipesPipe_source_sqs_target_lambda_function - PASS
TestAccPipesPipe_source_sqs_target_redshift - PASS
TestAccPipesPipe_source_sqs_target_step_function - PASS
TestAccPipesPipe_source_sqs_target_ecs_task
TestAccPipesPipe_targetParameters_inputTemplate - PASS
...

* Add support for Eventbridge Enrichment Parameters * Add support for Eventbridge Target Parameters * Add support for Eventbridge Source Parameters * Due to the amount of parameters add unit tests to ensure the mapping is correct to the right struct value * Update docs to support parameters

github-actions · 2023-05-26T21:18:43Z

Community Note

Voting for Prioritization

Please vote on this pull request by adding a 👍 reaction to the original post to help the community and maintainers prioritize this pull request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

For Submitters

Review the contribution guide relating to the type of change you are making to ensure all of the necessary steps have been taken.
For new resources and data sources, use skaff to generate scaffolding with comments detailing common expectations.
Whether or not the branch has been rebased will not impact prioritization, but doing so is always a welcome surprise.

github-actions · 2023-05-26T21:19:08Z

Thank you for your contribution! 🚀

Please note that typically Go dependency changes are handled in this repository by dependabot or the maintainers. This is to prevent pull request merge conflicts and further delay reviews of contributions. Remove any changes to the go.mod or go.sum files and commit them into this pull request.

Additional details:

Check open pull requests with the dependencies label to view other dependency updates.
If this pull request includes an update the AWS Go SDK (or any other dependency) version, only updates submitted via dependabot will be merged. This pull request will need to remove these changes and will need to be rebased after the existing dependency update via dependabot has been merged for this pull request to be reviewed.
If this pull request is for supporting a new AWS service:
- Ensure the new AWS service changes are following the Contributing Guide section on new services, in particular that the dependency addition and initial provider support are in a separate pull request from other changes (e.g. new resources). Contributions not following this item will not be reviewed until the changes are split.
- If this pull request is already a separate pull request from the above item, you can ignore this message.

internal/service/pipes/source_parameters.go

# Conflicts: # go.mod

This reverts commit 270783e.

This reverts commit 3d5cef3.

….sqs_queue_parameters'.

…ameters.dynamodb_stream_parameters'.

…ameters.activemq_broker_parameters'.

…arget'. Acceptance test output: % make testacc TESTARGS='-run=TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget' PKG=pipes ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./internal/service/pipes/... -v -count 1 -parallel 20 -run=TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget -timeout 180m === RUN TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget === PAUSE TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget === CONT TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget --- PASS: TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget (75.22s) PASS ok github.com/hashicorp/terraform-provider-aws/internal/service/pipes 80.505s

…get'. Acceptance test output: % make testacc TESTARGS='-run=TestAccPipesPipe_activeMQSourceStepFunctionTarget' PKG=pipes ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./internal/service/pipes/... -v -count 1 -parallel 20 -run=TestAccPipesPipe_activeMQSourceStepFunctionTarget -timeout 180m === RUN TestAccPipesPipe_activeMQSourceStepFunctionTarget === PAUSE TestAccPipesPipe_activeMQSourceStepFunctionTarget === CONT TestAccPipesPipe_activeMQSourceStepFunctionTarget --- PASS: TestAccPipesPipe_activeMQSourceStepFunctionTarget (1084.58s) PASS ok github.com/hashicorp/terraform-provider-aws/internal/service/pipes 1090.076s

Acceptance test output: % make testacc TESTARGS='-run=TestAccPipesPipe_rabbitMQSourceEventBusTarget' PKG=pipes ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./internal/service/pipes/... -v -count 1 -parallel 20 -run=TestAccPipesPipe_rabbitMQSourceEventBusTarget -timeout 180m === RUN TestAccPipesPipe_rabbitMQSourceEventBusTarget === PAUSE TestAccPipesPipe_rabbitMQSourceEventBusTarget === CONT TestAccPipesPipe_rabbitMQSourceEventBusTarget --- PASS: TestAccPipesPipe_rabbitMQSourceEventBusTarget (718.07s) PASS ok github.com/hashicorp/terraform-provider-aws/internal/service/pipes 723.681s

Acceptance test output: % make testacc TESTARGS='-run=TestAccPipesPipe_sqsSourceRedshiftTarget' PKG=pipes ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./internal/service/pipes/... -v -count 1 -parallel 20 -run=TestAccPipesPipe_sqsSourceRedshiftTarget -timeout 180m === RUN TestAccPipesPipe_sqsSourceRedshiftTarget === PAUSE TestAccPipesPipe_sqsSourceRedshiftTarget === CONT TestAccPipesPipe_sqsSourceRedshiftTarget --- PASS: TestAccPipesPipe_sqsSourceRedshiftTarget (316.69s) PASS ok github.com/hashicorp/terraform-provider-aws/internal/service/pipes 323.496s

…t' PKG=pipes ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./internal/service/pipes/... -v -count 1 -parallel 20 -run=TestAccPipesPipe_sqsSourceBatchJobTarget -timeout 180m === RUN TestAccPipesPipe_sqsSourceBatchJobTarget === PAUSE TestAccPipesPipe_sqsSourceBatchJobTarget === CONT TestAccPipesPipe_sqsSourceBatchJobTarget --- PASS: TestAccPipesPipe_sqsSourceBatchJobTarget (164.41s) PASS ok github.com/hashicorp/terraform-provider-aws/internal/service/pipes 170.141s

Acceptance test output: % make testacc TESTARGS='-run=TestAccPipesPipe_enrichmentParameters' PKG=pipes ==> Checking that code complies with gofmt requirements... TF_ACC=1 go test ./internal/service/pipes/... -v -count 1 -parallel 20 -run=TestAccPipesPipe_enrichmentParameters -timeout 180m === RUN TestAccPipesPipe_enrichmentParameters === PAUSE TestAccPipesPipe_enrichmentParameters === CONT TestAccPipesPipe_enrichmentParameters --- PASS: TestAccPipesPipe_enrichmentParameters (140.48s) PASS ok github.com/hashicorp/terraform-provider-aws/internal/service/pipes 146.365s

…k lines [Expected: 1; Actual: 2]'.

ewbankkit

LGTM 🚀.

% make testacc TESTARGS='-run=TestAccPipesPipe_' PKG=pipes ACCTEST_PARALLELISM=3
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./internal/service/pipes/... -v -count 1 -parallel 3  -run=TestAccPipesPipe_ -timeout 180m
=== RUN   TestAccPipesPipe_basicSQS
=== PAUSE TestAccPipesPipe_basicSQS
=== RUN   TestAccPipesPipe_disappears
=== PAUSE TestAccPipesPipe_disappears
=== RUN   TestAccPipesPipe_description
=== PAUSE TestAccPipesPipe_description
=== RUN   TestAccPipesPipe_desiredState
=== PAUSE TestAccPipesPipe_desiredState
=== RUN   TestAccPipesPipe_enrichment
=== PAUSE TestAccPipesPipe_enrichment
=== RUN   TestAccPipesPipe_enrichmentParameters
=== PAUSE TestAccPipesPipe_enrichmentParameters
=== RUN   TestAccPipesPipe_sourceParameters_filterCriteria
=== PAUSE TestAccPipesPipe_sourceParameters_filterCriteria
=== RUN   TestAccPipesPipe_nameGenerated
=== PAUSE TestAccPipesPipe_nameGenerated
=== RUN   TestAccPipesPipe_namePrefix
=== PAUSE TestAccPipesPipe_namePrefix
=== RUN   TestAccPipesPipe_roleARN
=== PAUSE TestAccPipesPipe_roleARN
=== RUN   TestAccPipesPipe_tags
=== PAUSE TestAccPipesPipe_tags
=== RUN   TestAccPipesPipe_targetUpdate
=== PAUSE TestAccPipesPipe_targetUpdate
=== RUN   TestAccPipesPipe_targetParameters_inputTemplate
=== PAUSE TestAccPipesPipe_targetParameters_inputTemplate
=== RUN   TestAccPipesPipe_kinesisSourceAndTarget
=== PAUSE TestAccPipesPipe_kinesisSourceAndTarget
=== RUN   TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget
=== PAUSE TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget
=== RUN   TestAccPipesPipe_activeMQSourceStepFunctionTarget
=== PAUSE TestAccPipesPipe_activeMQSourceStepFunctionTarget
=== RUN   TestAccPipesPipe_rabbitMQSourceEventBusTarget
=== PAUSE TestAccPipesPipe_rabbitMQSourceEventBusTarget
=== RUN   TestAccPipesPipe_mskSourceHTTPTarget
    acctest.go:79: DependencyViolation errors deleting subnets and security group
--- SKIP: TestAccPipesPipe_mskSourceHTTPTarget (0.00s)
=== RUN   TestAccPipesPipe_selfManagedKafkaSourceLambdaFunctionTarget
    acctest.go:79: DependencyViolation errors deleting subnets and security group
--- SKIP: TestAccPipesPipe_selfManagedKafkaSourceLambdaFunctionTarget (0.00s)
=== RUN   TestAccPipesPipe_sqsSourceRedshiftTarget
=== PAUSE TestAccPipesPipe_sqsSourceRedshiftTarget
=== RUN   TestAccPipesPipe_SourceSageMakerTarget
    acctest.go:79: aws_sagemaker_pipeline resource not yet implemented
--- SKIP: TestAccPipesPipe_SourceSageMakerTarget (0.00s)
=== RUN   TestAccPipesPipe_sqsSourceBatchJobTarget
=== PAUSE TestAccPipesPipe_sqsSourceBatchJobTarget
=== RUN   TestAccPipesPipe_sqsSourceECSTaskTarget
    acctest.go:79: ValidationException: [numeric instance is lower than the required minimum (minimum: 1, found: 0)]
--- SKIP: TestAccPipesPipe_sqsSourceECSTaskTarget (0.00s)
=== CONT  TestAccPipesPipe_basicSQS
=== CONT  TestAccPipesPipe_tags
=== CONT  TestAccPipesPipe_enrichmentParameters
--- PASS: TestAccPipesPipe_basicSQS (106.62s)
=== CONT  TestAccPipesPipe_roleARN
--- PASS: TestAccPipesPipe_tags (142.38s)
=== CONT  TestAccPipesPipe_namePrefix
--- PASS: TestAccPipesPipe_enrichmentParameters (158.63s)
=== CONT  TestAccPipesPipe_nameGenerated
--- PASS: TestAccPipesPipe_roleARN (132.92s)
=== CONT  TestAccPipesPipe_sourceParameters_filterCriteria
--- PASS: TestAccPipesPipe_nameGenerated (84.64s)
=== CONT  TestAccPipesPipe_activeMQSourceStepFunctionTarget
--- PASS: TestAccPipesPipe_namePrefix (100.89s)
=== CONT  TestAccPipesPipe_sqsSourceBatchJobTarget
--- PASS: TestAccPipesPipe_sqsSourceBatchJobTarget (170.14s)
=== CONT  TestAccPipesPipe_sqsSourceRedshiftTarget
--- PASS: TestAccPipesPipe_sourceParameters_filterCriteria (273.14s)
=== CONT  TestAccPipesPipe_rabbitMQSourceEventBusTarget
--- PASS: TestAccPipesPipe_sqsSourceRedshiftTarget (299.32s)
=== CONT  TestAccPipesPipe_kinesisSourceAndTarget
--- PASS: TestAccPipesPipe_kinesisSourceAndTarget (76.95s)
=== CONT  TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget
--- PASS: TestAccPipesPipe_dynamoDBSourceCloudWatchLogsTarget (82.84s)
=== CONT  TestAccPipesPipe_desiredState
--- PASS: TestAccPipesPipe_desiredState (189.01s)
=== CONT  TestAccPipesPipe_enrichment
--- PASS: TestAccPipesPipe_enrichment (171.56s)
=== CONT  TestAccPipesPipe_description
--- PASS: TestAccPipesPipe_activeMQSourceStepFunctionTarget (1038.32s)
=== CONT  TestAccPipesPipe_disappears
--- PASS: TestAccPipesPipe_rabbitMQSourceEventBusTarget (779.00s)
=== CONT  TestAccPipesPipe_targetParameters_inputTemplate
--- PASS: TestAccPipesPipe_disappears (86.23s)
=== CONT  TestAccPipesPipe_targetUpdate
--- PASS: TestAccPipesPipe_description (209.32s)
--- PASS: TestAccPipesPipe_targetParameters_inputTemplate (160.39s)
--- PASS: TestAccPipesPipe_targetUpdate (145.67s)
PASS
ok  	github.com/hashicorp/terraform-provider-aws/internal/service/pipes	1519.376s

ewbankkit · 2023-06-26T22:15:23Z

@jordanfinners Thanks for the contribution 🎉 👏.
I too was unable to get all the acceptance tests passing and have added Skip calls in those that I couldn't get cleanly passing.
I added a note to the CHANGELOG entries for this change calling out that we couldn't test all cases.

I changed the names of some of the nested parameters blocks to match the underlying AWS API.

github-actions · 2023-06-30T01:51:39Z

This functionality has been released in v5.6.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2023-07-31T02:11:35Z

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

This was referenced May 30, 2023

[Bug]: ValidationException: {"ValidationErrorFields":[{"Name":"SourceParameters.DynamoDBStreamParameters","Message":"Missing required parameter."}]} #30773

Closed

[Enhancement]: EventBridge Pipes #28153

Closed

justinretzolk added enhancement Requests to existing resources that expand the functionality or scope. and removed needs-triage Waiting for first response or review from a maintainer. labels Jun 1, 2023

joedarby reviewed Jun 7, 2023

View reviewed changes

internal/service/pipes/source_parameters.go Outdated Show resolved Hide resolved

ewbankkit added 16 commits June 9, 2023 12:24

Merge branch 'main' into HEAD

8794d92

# Conflicts: # go.mod

r/aws_pipes_pipe: Alphabetize attributes.

e80db56

Add CHANGELOG entry.

76d73ea

Tidy up 'enrichment_parameters'.

cc443f8

Tidy up 'source_parameters'.

bc29dd0

Tidy up 'target_parameters'.

24b9f39

'sage_maker_pipeline' -> 'sagemaker_pipeline'.

3229613

pipes: Use 'go-cmp' in unit tests.

3d5cef3

pipes: Use 'reflect.DeepEquals' in unit tests.

270783e

Revert "pipes: Use 'reflect.DeepEquals' in unit tests."

10324c7

This reverts commit 270783e.

Revert "pipes: Use 'go-cmp' in unit tests."

bc9e9a3

This reverts commit 3d5cef3.

r/aws_pipes_pipe: 'enrichment_parameters.http_parameters' changes.

e93fbba

r/aws_pipes_pipe: 'source_parameters.filter_criteria' changes.

1299833

r/aws_pipes_pipe: 'source_parameters.sqs_queue' -> 'source_parameters…

f67104c

….sqs_queue_parameters'.

r/aws_pipes_pipe: 'source_parameters.dynamo_db_stream' -> 'source_par…

fc4286f

…ameters.dynamodb_stream_parameters'.

r/aws_pipes_pipe: 'source_parameters.active_mq_broker' -> 'source_par…

3e6bfec

…ameters.activemq_broker_parameters'.

ewbankkit added 7 commits June 24, 2023 15:50

Add 'TestAccPipesPipe_mskSourceHTTPTarget'.

b3bd872

Add 'TestAccPipesPipe_selfManagedKafkaSourceLambdaFunctionTarget'.

2181c3e

Add 'TestAccPipesPipe_sqsSourceSagemakerTarget'.

77bd4d5

ewbankkit mentioned this pull request Jun 26, 2023

Support Amazon SageMaker Pipelines #32225

Closed

ewbankkit added 3 commits June 26, 2023 14:44

Add 'TestAccPipesPipe_sqsSourceECSTaskTarget'.

80b2a9b

github-actions bot added the generators Relates to code generators. label Jun 26, 2023

ewbankkit added 7 commits June 26, 2023 17:17

Fix terrafmt errors in acceptance test configurations.

622cb2a

Fix markdown-lint 'MD012/no-multiple-blanks Multiple consecutive blan…

db237c6

…k lines [Expected: 1; Actual: 2]'.

Fix semgrep errors.

4d08912

Add NOTE to CHANGELOG entries.

517e8a2

Fix golangci-lint 'paralleltest'.

bbb25ff

Fix golangci-lint 'unused'.

a2c7ca6

Fix golangci-lint 'staticcheck'.

969b6a4

ewbankkit approved these changes Jun 26, 2023

View reviewed changes

ewbankkit merged commit 096459a into hashicorp:main Jun 26, 2023

github-actions bot added this to the v5.6.0 milestone Jun 26, 2023

jordanfinners deleted the f-pipes-update branch June 30, 2023 18:11

NixM0nk3y mentioned this pull request Jul 9, 2023

use correct parameter key in ECS target in aws_pipes_pipe #32432

Merged

This was referenced Jul 19, 2023

[Bug]: 'Invalid address to set' when using an ECS task as target of an EventBridge pipe #32586

Closed

[Bug]: Cannot update EventBridge Pipe in place when source is DynamoDB Stream #32423

Closed

github-actions bot locked as resolved and limited conversation to collaborators Jul 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Eventbridge Pipes parameters #31607

Support Eventbridge Pipes parameters #31607

jordanfinners commented May 26, 2023 •

edited by ewbankkit

Loading

github-actions bot commented May 26, 2023

github-actions bot commented May 26, 2023

ewbankkit left a comment

ewbankkit commented Jun 26, 2023

github-actions bot commented Jun 30, 2023

github-actions bot commented Jul 31, 2023

Support Eventbridge Pipes parameters #31607

Support Eventbridge Pipes parameters #31607

Conversation

jordanfinners commented May 26, 2023 • edited by ewbankkit Loading

Description

Relations

Output from Acceptance Testing

github-actions bot commented May 26, 2023

Community Note

github-actions bot commented May 26, 2023

ewbankkit left a comment

Choose a reason for hiding this comment

ewbankkit commented Jun 26, 2023

github-actions bot commented Jun 30, 2023

github-actions bot commented Jul 31, 2023

jordanfinners commented May 26, 2023 •

edited by ewbankkit

Loading