Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix generate params regions lookup + refactor pipeline management #584

Merged
merged 4 commits into from
Jan 1, 2023

Conversation

sbkok
Copy link
Collaborator

@sbkok sbkok commented Dec 23, 2022

Why?

Opening this PR with four different changes that are heavily related.
Splitting these into different PRs would consume too much time unfortunately.
So please bear with me:

1. Generate params region lookup

  • With the changes introduced in PR Adding deployment map source to SSM params used to identify out of date pipelines #525, support was added for different deployment map providers.

  • This changed the location of the pipeline parameters to the scope of the source. In the case of our S3 deployment map provider, that changed: /deployment/${pipeline_name}/regions to /deployment/S3/${pipeline_name}/regions.

  • In the generate_params.py helper, it still continued to look for the regions in the original SSM Parameter Store location.

2. Support debug logs in generate_params.py

  • We need more debug information to enable debugging generate_params.py efficiently.

3. Merge in default deployment props into target specifications

  • The default provider, properties, and regions were not merged into the target configuration yet.
  • This moved the lookup of the defaults to the execution step of the pipeline deployment.
  • If we move applying the defaults to the input stage, this makes it easier to see what will be applied at execution level. As that input is captured in the new Pipeline Management state machine.

4. S3 Extract property default not defined

  • The S3 Deployment provider should set the Extract property to False when not set, as specified in our docs. It did not.

What?

1. Generate params region lookup

  • Refactored pipeline management to separate execution details from the pipeline definition, the pipeline input, and the SSM parameters retrieved.
    • Renaming input to pipeline_input to clearly indicate what input we are referring to.
  • Refactored line length issues spotted while making changes to forward the deployment map source and name.
  • Added the deployment_map_source and deployment_map_name context to the ADF Pipeline Management state machine.
  • Fixed an issue in the pipeline generation, where the EventBridge rule would be defined if the account_id would be set when CodeCommit was not used as the source provider.
  • Removed ${bootstrap_repo}/adf-build/shared/cdk/generate_pipeline_inputs.py as that is replaced by the ADF Pipeline Management using the Step Function state machine instead.
  • Moved flattening lists to the list_utils helper, making it easier to reuse this logic elsewhere. Since the lists had to be sorted and made unique each time this logic was moved to this function too. Reducing repeated code and different variations trying to achieve the same thing.

2. Support debug logs in generate_params.py

  • Added debug log messages.
  • End-users can add the ADF_LOG_LEVEL environment variable and set it to DEBUG to get the logs printed.

3. Merge in default deployment props into target specifications

  • Apply default deploy provider configuration to the target configuration at input generation time.

4. S3 Extract property default not defined

  • Obviously setting the default to False :D

By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.

@sbkok sbkok added this to the v3.2.0 milestone Dec 23, 2022
**Why?**

With the changes introduced in PR awslabs#525, support was added for different
deployment map providers.

This changed the location of the pipeline parameters to the scope of the
source. In the case of our S3 deployment map provider, that changed:
`/deployment/${pipeline_name}/regions` to
`/deployment/S3/${pipeline_name}/regions`.

In the `generate_params.py` helper, it still continued to look for the regions
in the original SSM Parameter Store location.

**What?**

* Refactored pipeline management to separate execution details from the
  pipeline definition, the pipeline input, and the SSM parameters retrieved.
  * Renaming `input` to `pipeline_input` to clearly indicate what input we are
    referring to.
* Refactored line length issues spotted while making changes to forward the
  deployment map source and name.
* Added the `deployment_map_source` and `deployment_map_name` context to the
  ADF Pipeline Management state machine.
* Fixed an issue in the pipeline generation, where the EventBridge rule would
  be defined if the account_id would be set when CodeCommit was not used as
  the source provider.
* Removed
  `${bootstrap_repo}/adf-build/shared/cdk/generate_pipeline_inputs.py` as that
  is replaced by the ADF Pipeline Management using the Step Function state
  machine instead.
* Moved flattening lists to the list_utils helper, making it easier to reuse
  this logic elsewhere. Since the lists had to be sorted and made unique each
  time this logic was moved to this function too. Reducing repeated code and
  different variations trying to achieve the same thing.
**Why?**

We need more debug information to enable debugging generate params efficiently.

**What?**

* Added debug log levels.
* End-users can add the `ADF_LOG_LEVEL` environment variable and set it to
  `DEBUG` to get the logs printed.
**Why?**

The default provider, properties, and regions were not merged into
the target configuration yet.

This moved the lookup of the defaults to the execution step of the
pipeline deployment.

If we move applying the defaults to the input stage, this makes it easier
to see what will be applied at execution level. As that input is captured
in the new Pipeline Management state machine.

**What?**

* Apply default deploy provider configuration to the target configuration at
  input generation time.
@sbkok sbkok force-pushed the fix/generate-params-regions-lookup branch from a5e98c9 to 44c1dae Compare December 23, 2022 17:39
@StewartW StewartW merged commit a4de9ab into awslabs:master Jan 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants