Add FluxCD support for S3 Bucket sources #1007

jkataja · 2024-05-29T00:03:30Z

Issue #, if available:

Description of changes:

implement support for FluxCD Bucket source: https://fluxcd.io/flux/components/source/buckets/
bump FluxCD default version to 2.13.0
use the more reliable OCI registry for the FluxCD Helm chart
make indentation consistent within the modified files
changed typo in defaultKustomiationProps
basic use case tested
kustomizations not tested

Example use:

        .addOns(new blueprints.addons.FluxCDAddOn({
            buckets: [
                {
                    name: "bootstrap-bucket",
                    bucketName:base.bucket.bucketName,
                    bucketRegion: cdk.Aws.REGION,
                }
            ],
        }),

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

elamaran11

Approach LGTM. @shapirov103 Pls check.

jkataja · 2024-06-04T17:31:42Z

@shapirov103 Hi, do you require any inputs from me to test and review the pull request? For example how I setup the Flux CD source bucket.

shapirov103

@jkataja thank you for this contribution, looks good overall, I added a couple of design/maintenance level comments.

As far as testing, please share an example that you used for validation. We need to run it manually in this case as flux is not continuously tested with the end-to-end tests.

shapirov103 · 2024-06-12T21:44:43Z

lib/addons/fluxcd/index.ts

+    kustomizations?: FluxKustomizationProps[];
+}
+
+/**


As far as design - therepository construct is effectively misused as a vehicle to carry bucket info. Consider a case when we add a URL validator (lack of it is a miss) to the repoUrl and start failing valid cases for the bucket name.
why not omit this structure from the type and introduce a clear bucket type?

see commit 1c95b13

test bucket construct changes

shapirov103 · 2024-06-12T21:47:03Z

lib/addons/fluxcd/index.ts

+    chart: "flux2",
+    version: "2.13.0",
+    release: "blueprints-fluxcd-addon",
+    repository: "oci://ghcr.io/fluxcd-community/charts/flux2",


with the switch to OCR, are you aware of a mechanism for us to discover when new versions of flux become available (other than manual look up)?

seems that gh release will show it but the output is not strictly for this chart:

$ gh release list --repo fluxcd-community/helm-charts flux2-sync-1.9.0 Latest (flux2-sync-1.9.0) about 1 month ago flux2-notification-1.15.0 (flux2-notification-1.15.0) about 1 month ago flux2-2.13.0 (flux2-2.13.0) about 1 month ago flux2-2.12.4 (flux2-2.12.4) about 3 months ago flux2-sync-1.8.2 (flux2-sync-1.8.2) about 3 months ago flux2-notification-1.14.1 (flux2-notification-1.14.1) about 3 months ago flux2-2.12.3 (flux2-2.12.3) about 3 months ago flux2-notification-1.14.0 (flux2-notification-1.14.0) about 4 months ago flux2-sync-1.8.1 (flux2-sync-1.8.1) about 5 months ago flux2-notification-1.13.1 (flux2-notification-1.13.1) about 5 months ago flux2-2.12.2 (flux2-2.12.2) about 5 months ago flux2-2.12.1 (flux2-2.12.1) about 6 months ago flux2-sync-1.8.0 (flux2-sync-1.8.0) about 6 months ago flux2-notification-1.13.0 (flux2-notification-1.13.0) about 6 months ago flux2-2.12.0 (flux2-2.12.0) about 6 months ago flux2-2.11.1 (flux2-2.11.1) about 7 months ago flux2-sync-1.7.3 (flux2-sync-1.7.3) about 7 months ago flux2-notification-1.12.4 (flux2-notification-1.12.4) about 7 months ago flux2-multi-tenancy-0.0.6 (flux2-multi-tenancy-0.0.6) about 7 months ago flux2-2.11.0 (flux2-2.11.0) about 7 months ago flux2-2.10.6 (flux2-2.10.6) about 7 months ago flux2-sync-1.7.2 (flux2-sync-1.7.2) about 7 months ago flux2-notification-1.12.3 (flux2-notification-1.12.3) about 7 months ago flux2-2.10.5 (flux2-2.10.5) about 7 months ago flux2-notification-1.12.2 (flux2-notification-1.12.2) about 7 months ago flux2-multi-tenancy-0.0.5 (flux2-multi-tenancy-0.0.5) about 7 months ago flux2-2.10.4 (flux2-2.10.4) about 7 months ago flux2-2.10.3 (flux2-2.10.3) about 7 months ago flux2-2.10.2 (flux2-2.10.2) about 8 months ago flux2-sync-1.7.1 (flux2-sync-1.7.1) about 8 months ago

commented in 455a6a3

jkataja · 2024-06-14T18:46:12Z

@jkataja thank you for this contribution, looks good overall, I added a couple of design/maintenance level comments.

You're welcome!

As far as testing, please share an example that you used for validation. We need to run it manually in this case as flux is not continuously tested with the end-to-end tests.

Sure, so I create the bucket in a separate stack using:

new s3.Bucket(this, "SourceBucket", {
      removalPolicy: cdk.RemovalPolicy.RETAIN,
      objectOwnership: s3.ObjectOwnership.BUCKET_OWNER_ENFORCED,
    });

Allow the node role to access the bucket by using:

        base.bucket.grantRead(role);

And then initialize the FluxCD addon as follows:

    .addOns(new blueprints.addons.FluxCDAddOn({
        buckets: [
            {
                name: "bootstrap-bucket",
                bucketName:base.bucket.bucketName,
                bucketRegion: cdk.Aws.REGION,
            }
        ],
    }),
...

~~Contains untested changes.~~ Edit: tested basic deployment without Kustomizations

@shapirov103 please take another look of the pull request, I have addressed the comments and tested manually.

jkataja · 2024-06-15T12:10:20Z

The pipeline started failing with the eksworkshop.com returning HTTP 403 with AccessDenied response from S3

shapirov103

LGTM

shapirov103 · 2024-06-19T04:54:06Z

/do-e2e-tests

aws-ia-ci

end to end tests passed

elamaran11 · 2024-06-19T11:52:27Z

@jkataja Please merge from main, that should fix the GH Action errors.

- bump FluxCD default version to 2.13.0 - use the more reliable OCI registry for the FluxCD Helm chart

jkataja · 2024-06-19T13:37:00Z

@elamaran11 I rebased on main, please try again

elamaran11 · 2024-06-19T13:51:04Z

/do-e2e-tests

aws-ia-ci

end to end tests passed

elamaran11 · 2024-06-19T14:58:07Z

@shapirov103 LG to merge.

jkataja requested review from aws-ia-ci, parkand1, shapirov103, youngjeong46, elamaran11 and a team as code owners May 29, 2024 00:03

elamaran11 reviewed May 29, 2024

View reviewed changes

jkataja force-pushed the feature/fluxcd-bucket-source branch from f8744e2 to 7cef337 Compare June 12, 2024 20:22

shapirov103 reviewed Jun 12, 2024

View reviewed changes

jkataja force-pushed the feature/fluxcd-bucket-source branch 2 times, most recently from 2ef8562 to c80694d Compare June 15, 2024 11:56

shapirov103 approved these changes Jun 19, 2024

View reviewed changes

shapirov103 requested a review from elamaran11 June 19, 2024 04:53

aws-ia-ci approved these changes Jun 19, 2024

View reviewed changes

jkataja added 5 commits June 19, 2024 15:35

add FluxCD support for S3 Bucket sources

88d832b

- bump FluxCD default version to 2.13.0 - use the more reliable OCI registry for the FluxCD Helm chart

add package update instructions

98cecd3

end misuse of repository to carry the bucket

71f6d0b

fix comment for the default bucket namespace

6354fee

remove default namespace in bucket

031c2e5

jkataja force-pushed the feature/fluxcd-bucket-source branch from c80694d to 031c2e5 Compare June 19, 2024 13:35

aws-ia-ci approved these changes Jun 19, 2024

View reviewed changes

shapirov103 merged commit 7f38bbc into aws-quickstart:main Jun 19, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FluxCD support for S3 Bucket sources #1007

Add FluxCD support for S3 Bucket sources #1007

jkataja commented May 29, 2024 •

edited

Loading

elamaran11 left a comment

jkataja commented Jun 4, 2024

shapirov103 left a comment

shapirov103 Jun 12, 2024

jkataja Jun 14, 2024 •

edited

Loading

shapirov103 Jun 12, 2024

jkataja Jun 14, 2024

jkataja commented Jun 14, 2024 •

edited

Loading

jkataja commented Jun 15, 2024

shapirov103 left a comment

shapirov103 commented Jun 19, 2024

aws-ia-ci left a comment

elamaran11 commented Jun 19, 2024

jkataja commented Jun 19, 2024

elamaran11 commented Jun 19, 2024

aws-ia-ci left a comment

elamaran11 commented Jun 19, 2024

Add FluxCD support for S3 Bucket sources #1007

Add FluxCD support for S3 Bucket sources #1007

Conversation

jkataja commented May 29, 2024 • edited Loading

elamaran11 left a comment

Choose a reason for hiding this comment

jkataja commented Jun 4, 2024

shapirov103 left a comment

Choose a reason for hiding this comment

shapirov103 Jun 12, 2024

Choose a reason for hiding this comment

jkataja Jun 14, 2024 • edited Loading

Choose a reason for hiding this comment

shapirov103 Jun 12, 2024

Choose a reason for hiding this comment

jkataja Jun 14, 2024

Choose a reason for hiding this comment

jkataja commented Jun 14, 2024 • edited Loading

jkataja commented Jun 15, 2024

shapirov103 left a comment

Choose a reason for hiding this comment

shapirov103 commented Jun 19, 2024

aws-ia-ci left a comment

Choose a reason for hiding this comment

elamaran11 commented Jun 19, 2024

jkataja commented Jun 19, 2024

elamaran11 commented Jun 19, 2024

aws-ia-ci left a comment

Choose a reason for hiding this comment

elamaran11 commented Jun 19, 2024

jkataja commented May 29, 2024 •

edited

Loading

jkataja Jun 14, 2024 •

edited

Loading

jkataja commented Jun 14, 2024 •

edited

Loading