Refactor: Separate Pipeline Management into Serverless Application #424

StewartW · 2022-01-10T11:38:21Z

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Seperate out Pipeline Mananagement away from CodePipeline and remove the major bottlenecks of building each deployment map sequentially. Starts to isolate the CodeCOmmit repository and moves the source of deployment maps into an S3 Bucket. Sets the groundwork for future refactoring of Pipeline Management and moves towards enabling decentralised deployment maps.

…into refactor/pipeline-management

sbkok

Thanks for refactoring this logic! Very valuable!

I noticed a few changes were missing in the logic. For example wave generation and another PR that I linked in the review comments.
Could you make sure the changes introduced by those PRs are also included in this PR?

docs/images/technical-diagrams.drawio

docs/technical-guide.md

...repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/create_repository.py

...ambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/deployment_map.py

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/repo.py

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/rule.py

src/template.yml

Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>

…s-deployment-framework into refactor/pipeline-management

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/target.py

…into refactor/pipeline-management

...ase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_deployment_map.py

src/lambda_codebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/global.yml

...ambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/cloudformation.py

…into refactor/pipeline-management

Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>

…repository/adf-bootstrap/deployment/global.yml

…s-deployment-framework into refactor/pipeline-management

sbkok

Great improvements, thanks for making those.
I've got a number of minor suggested changes, so we are close to merge this too!

...repository/adf-bootstrap/deployment/lambda_codebase/pipeline_management/create_repository.py

...ory/adf-bootstrap/deployment/lambda_codebase/pipeline_management/generate_pipeline_inputs.py

...f-bootstrap/deployment/lambda_codebase/pipeline_management/identify_out_of_date_pipelines.py

...odebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml

...ambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/deployment_map.py

...ase/initial_commit/bootstrap_repository/adf-build/shared/python/tests/test_deployment_map.py

Adding ADF Prefix to make it clearer that they are part of the ADF ecosystem Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>

...odebase/initial_commit/bootstrap_repository/adf-bootstrap/deployment/pipeline_management.yml

...ambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/deployment_map.py

…s-deployment-framework into refactor/pipeline-management

sbkok

Thanks for making all these changes @StewartW!
This will help speed up ADF a lot, excellent work!

javydekoning

This looks good to me!

I suggest also storing the .drawio originals to the repo so the diagram can be easily updated when there are changes to the statemachine later.

sbkok · 2022-08-26T16:25:47Z

For reference, the .drawio.png files that were added by the PR also include the source of the image.
You can edit them by opening the png file directly with the DrawIO tool.

@javydekoning

According to @javydekoning, it's probably left over from awslabs#424

@javydekoning

* Update ADF pipeline generation to CDKv2 * Add trailing commas to imports Co-authored-by: Simon Kok <sbkok@users.noreply.github.com> * Apply suggestions from code review * Fix merge * Upgrade to latest CDK + SAM CLI, Python 3.10 (Py3.11 in CodeBuild) **Why?** As this commit will update the new default CodeBuild container image to use, we will pick the most recent Standard CodeBuild image that is available at the time of this commit. Which is STANDARD_7_0. This allows us to continue to support the version for the foreseeable future. As we rather not need to change this frequently, as that would introduce a breaking change. * Remove cdk package as it's no longer used According to @javydekoning, it's probably left over from #424 * Use aws-cdk package instead of cdk --------- Co-authored-by: Simon Kok <sbkok@users.noreply.github.com> Co-authored-by: Javy de Koning <javydekoning+github@gmail.com> Co-authored-by: Simon Kok <mail@simonkok.com>

StewartW force-pushed the refactor/pipeline-management branch from 26a70ce to 26f2819 Compare April 12, 2022 10:10

StewartW force-pushed the refactor/pipeline-management branch from 26f2819 to 305ca55 Compare April 12, 2022 11:27

StewartW changed the title ~~Refactor pipeline management~~ Refactor: Separate Pipeline Management into Serverless Application Apr 12, 2022

Updating documentation

6d1451b

StewartW marked this pull request as ready for review April 25, 2022 14:05

StewartW requested review from sbkok and dsudduth April 25, 2022 14:06

StewartW added the enhancement New feature or request label May 20, 2022

StewartW added 4 commits August 12, 2022 10:00

Merge branch 'master' of github.com:awslabs/aws-deployment-framework …

2a7709c

…into refactor/pipeline-management

Merge branch 'master' of github.com:awslabs/aws-deployment-framework …

9e52c9a

…into refactor/pipeline-management

Fixing yamllint errors

6050abe

final tox issues:

7f9d1c9

sbkok requested changes Aug 16, 2022

View reviewed changes

StewartW and others added 4 commits August 16, 2022 13:02

Update docs/technical-guide.md

b3b7a10

Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>

Fixing some issues with testing

e3dae9b

Merge branch 'refactor/pipeline-management' of github.com:StewartW/aw…

1eb4088

…s-deployment-framework into refactor/pipeline-management

Enabling create repo based on config value

9fd1626

StewartW commented Aug 17, 2022

View reviewed changes

src/lambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/target.py Show resolved Hide resolved

StewartW added 4 commits August 17, 2022 17:23

Code Review Suggestions

88e6b0f

Merge branch 'master' of github.com:awslabs/aws-deployment-framework …

434b2e3

…into refactor/pipeline-management

Copy Paste Error

aef4b72

removing packaged.yml and fixing trailing spaces

1a8d7ea

StewartW force-pushed the refactor/pipeline-management branch 2 times, most recently from 5f27224 to 17767af Compare August 18, 2022 08:40

Missing default scm config value

29a4fee

StewartW force-pushed the refactor/pipeline-management branch from 17767af to 29a4fee Compare August 18, 2022 09:33

sbkok requested changes Aug 19, 2022

View reviewed changes

sbkok mentioned this pull request Aug 19, 2022

Fix: Wave deployment input data #484

Closed

Merge branch 'master' of github.com:awslabs/aws-deployment-framework …

9394231

…into refactor/pipeline-management

StewartW and others added 6 commits August 22, 2022 10:48

Apply suggestions from code review

56d45ec

Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>

Apply suggestions from code review

0115d29

Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>

Changing permissions of src/lambda_codebase/initial_commit/bootstrap_…

b9bbc9e

…repository/adf-bootstrap/deployment/global.yml

Merge branch 'refactor/pipeline-management' of github.com:StewartW/aw…

0fc5b50

…s-deployment-framework into refactor/pipeline-management

Passing thru log level param

7950409

Restricting IAM permissions to prefix

9e11f20

sbkok requested changes Aug 25, 2022

View reviewed changes

Updating Function names

612855e

Adding ADF Prefix to make it clearer that they are part of the ADF ecosystem Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>

sbkok requested changes Aug 26, 2022

View reviewed changes

sbkok reviewed Aug 26, 2022

View reviewed changes

...ambda_codebase/initial_commit/bootstrap_repository/adf-build/shared/python/deployment_map.py Outdated Show resolved Hide resolved

sbkok and others added 6 commits August 26, 2022 15:30

Apply suggestions from code review

d62332d

Code Review Suggestions

a0e442d

Merge branch 'refactor/pipeline-management' of github.com:StewartW/aw…

e7c7254

…s-deployment-framework into refactor/pipeline-management

removing condition

f3dafaa

Add wave generation test from awslabs#484

8f7323c

Make use of Temp.Directory to fetch the deployment map files

b46750f

sbkok approved these changes Aug 26, 2022

View reviewed changes

sbkok requested a review from javydekoning August 26, 2022 14:58

javydekoning approved these changes Aug 26, 2022

View reviewed changes

sbkok merged commit 0032fab into awslabs:master Aug 26, 2022

javydekoning mentioned this pull request May 15, 2023

Update CDK from v1 to v2 #619

Merged

pergardebrink added a commit to pergardebrink/aws-deployment-framework that referenced this pull request May 15, 2023

Remove cdk package as it's no longer used

20f0395

According to @javydekoning, it's probably left over from awslabs#424

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor: Separate Pipeline Management into Serverless Application #424

Refactor: Separate Pipeline Management into Serverless Application #424

StewartW commented Jan 10, 2022 •

edited

Loading

sbkok left a comment

sbkok left a comment

sbkok left a comment

javydekoning left a comment

sbkok commented Aug 26, 2022

Refactor: Separate Pipeline Management into Serverless Application #424

Refactor: Separate Pipeline Management into Serverless Application #424

Conversation

StewartW commented Jan 10, 2022 • edited Loading

sbkok left a comment

Choose a reason for hiding this comment

sbkok left a comment

Choose a reason for hiding this comment

sbkok left a comment

Choose a reason for hiding this comment

javydekoning left a comment

Choose a reason for hiding this comment

sbkok commented Aug 26, 2022

StewartW commented Jan 10, 2022 •

edited

Loading