Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/S3 provider - set default account id, bucket name and object name; support s3 notification based event trigger #782

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

danxie1999
Copy link

@danxie1999 danxie1999 commented Feb 14, 2025

Why?

As part of our enterprise deployment, we run three instances of ADF across three separate AWS Organizations: Development Organization, Integration Organization, and Production Organization. Each organization may have different S3 Source Provider Bucket names. To enable the reuse of deployment maps across these organizations, it is necessary to define the default 's3-source-bucket-name' in the respective 'adf-config' file of each organization.
Additionally, the current S3 provider in the deployment map does not support default values for the S3 bucket name, object name, and Account ID. By introducing these default values, it simplifies the definitions of deployment maps and centralizes the management of these properties.
Furthermore, using S3 notification events and EventBridge rules to trigger source pipelines is more efficient than AWS CodePipeline polling. AWS specifically recommends this event-based approach for detecting changes over traditional pipeline polling methods

What?

  1. The S3 provider has been updated to include default properties for the S3 bucket name, S3 object, and account ID of the bucket. These properties can now be set centrally and used throughout the application.
  2. A new property poll_for_changes had been added to the S3 provider, if set False, S3 notification events and triggers will be enabled instead of polling pipelines.
  3. A new function has been added to forward S3 events from other AWS accounts to the current account. This function sets up an event rule in the target account to capture S3 events and forwards them to the current account's event bus, enabling cross-account event processing.

These changes will improve the ADF's ability to manage Multi-Organization S3 bucket configurations, automate data processing pipelines, and integrate with S3 buckets across multiple AWS accounts.

…bucket properties in S3 provder

      2) Add s3 source pipeline trigger via s3 notification events function
      3) Add cross account s3 event rule fowarder function
@danxie1999 danxie1999 force-pushed the feat/adf-platform-upgrade branch from 8da201f to 3a6e08b Compare February 14, 2025 12:10
Copy link
Contributor

@AndyEfaa AndyEfaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. Looks already very good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants