Feat/S3 provider - set default account id, bucket name and object name; support s3 notification based event trigger #782
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why?
As part of our enterprise deployment, we run three instances of ADF across three separate AWS Organizations: Development Organization, Integration Organization, and Production Organization. Each organization may have different S3 Source Provider Bucket names. To enable the reuse of deployment maps across these organizations, it is necessary to define the default 's3-source-bucket-name' in the respective 'adf-config' file of each organization.
Additionally, the current S3 provider in the deployment map does not support default values for the S3 bucket name, object name, and Account ID. By introducing these default values, it simplifies the definitions of deployment maps and centralizes the management of these properties.
Furthermore, using S3 notification events and EventBridge rules to trigger source pipelines is more efficient than AWS CodePipeline polling. AWS specifically recommends this event-based approach for detecting changes over traditional pipeline polling methods
What?
poll_for_changes
had been added to the S3 provider, if setFalse
, S3 notification events and triggers will be enabled instead of polling pipelines.These changes will improve the ADF's ability to manage Multi-Organization S3 bucket configurations, automate data processing pipelines, and integrate with S3 buckets across multiple AWS accounts.