Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(artifacts): Consider artifacts from stages when triggering from a pipeline stage #4568

Conversation

sergio-quintero
Copy link
Contributor

Follow up to #4489

Triggering a pipeline from Pipeline A using the pipeline stage where Pipeline B uses BakeManifestStage or DeployManifestStage which rely on expectedArtifacts is broken. ArtifactUtils is filtering the expectedArtifacts needed in stages.

In 1.28, you were able to define expectedArtifacts in Pipeline B, and when you triggered from Pipeline A, Pipeline B was able to resolve the artifacts from Pipeline A. Passing artifacts between pipelines

… pipeline stage

Co-authored-by: Nemesis Osorio <nemesis211_6@hotmail.com>
@sergio-quintero sergio-quintero force-pushed the fix-expected-artifacts-pipeline-stage branch from fbcede6 to d5611bf Compare October 17, 2023 23:11
@dbyron-sf dbyron-sf requested a review from jervi October 17, 2023 23:43
Copy link
Contributor

@jervi jervi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Tagging @xibz as well, since you also showed some interest in this area

Copy link
Contributor

@xibz xibz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I have some very minor nit comments

def requiredArtifactIds = pipelineConfig.get("stages", []).collectMany { stage ->
def artifactIds = []

//DeployManifestStage in ResolveDeploySourceManifestTask using ManifestEvaluator.getManifestArtifact
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
//DeployManifestStage in ResolveDeploySourceManifestTask using ManifestEvaluator.getManifestArtifact
// DeployManifestStage in ResolveDeploySourceManifestTask using ManifestEvaluator.getManifestArtifact

nit

// BakeManifestStage in CreateBakeManifestTask using BakeManifestContext for Kustomize
artifactIds += stage.inputArtifact?.id ?: []

// BakeManifestStage in CreateBakeManifestTask using BakeManifestContext for Helm
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these comments accurate? What I mean is I understand that Helm bakes with inputArtifacts but other stages like BakeCloudFoundry also utilize this?

Can we make this more generic and say something more like

// Some bake manifest stages may populate the inputArtifacts array. So we need to look to for those ids as well

And we may be able to get rid of the other comments as they seem to be very specific but don't capture the whole picture

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe even explain why we are looking for these ids as opposed to the expected artifact. Would be good to just document here saying we dont know how consuming pipelines will utilize artifacts from the parent pipeline, so we rely on any input artifact as opposed to the expected artifact

// in ResolveDeploySourceManifestTask using ManifestEvaluator.getRequiredArtifacts
def requiredArtifactIds = pipelineConfig.get("stages", []).collectMany {
it.requiredArtifactIds ?: []
def requiredArtifactIds = pipelineConfig.get("stages", []).collectMany { stage ->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this variable? Can't we just use expectedArtifactIds here?

Copy link
Contributor

@edgarulg edgarulg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! but before merging it let's address all comments :)

@sergio-quintero
Copy link
Contributor Author

Closing this PR because we found more stages with different field names that were not considered in this PR. Adding a list of fields in DependentPipelineStarter is not maintainable. Also, if someone creates a plugin with a new Stage that utilizes expectedArtifacts, they can define any field name which will not be included in this class.

We'll send a new PR with a better fix for this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants