TEP-0056: Pipelines in Pipelines [Proposal]

[jerop]

TEP-0056: Pipelines in Pipelines

Today, users can define and execute Tasks and Custom Tasks
in Pipelines. In this TEP, we propose allowing users to define
and execute Pipelines in Pipelines, alongside Tasks and
Custom Tasks.

We previously described the motivation for this TEP in:

• TEP-0056: Pipelines in Pipelines #374
• TEP-0056: Pipelines in Pipelines - Add Use Case #498
• TEP-0056: Pipelines in Pipelines [Problem Statement] #698

In this change, we add an overview of the proposal which includes
the API changes to the specification and status of Pipelines.

[pritidesai]

/kind tep
/assign @afrittoli
/assign @lbernick
/assign @vdemeester

[pritidesai]

Following up on the nested pipeline discussion, it is expected in the issue tektoncd/pipeline#2134 (comment) to run a set of tasks from the same pipeline as part of the feature request. I envision this as a set of tasks guarded by a when expression expected to run as a sub-pipeline in which the pipelineRef will be referring to itself. We might not be able to support this in the first round of design/implementation but something to keep in mind for future.

[abayer]

@pritidesai I didn't necessarily read that as meaning "run tasks X and Y from this pipeline as a sub-pipeline" - that seems more to me like a combination of matrix and pipeline-in-pipeline to run a pipeline, either a pre-existing one via reference or an embedded one in the matrix PipelineTask configuration.

[pritidesai]

@pritidesai I didn't necessarily read that as meaning "run tasks X and Y from this pipeline as a sub-pipeline" - that seems more to me like a combination of matrix and pipeline-in-pipeline to run a pipeline, either a pre-existing one via reference or an embedded one in the matrix PipelineTask configuration.

I am reading it as more like run tasks X through Y from this pipeline as sub-pipeline against a list of workspaces. I have been trying to think whether would it be feasible to support workspace mappings in matrix. Matrix by design, supports looping based on the parameters 🤔.

[jerop]

@afrittoli @lbernick @pritidesai @vdemeester - we have updated the proposal, please take a look

cc @abayer

[lbernick]

Thanks Jerop!

My only comment is that it would be helpful to have some yaml examples for the alternatives; I think it would be easier to see why we're rejecting the alternatives.

[vdemeester]

So, this is not specific for this particular TEP, but sometime I think we dig too quickly into the code. For example, here, the type showed in alternative are "implementation" detail. The important part of the TEP should be how the API looks like and is used ; whatever the "magic" we have to do to make it work in our go program, is implementation imo.

[jerop]

@pritidesai @vdemeester - Thank you for the reviews! Updated the TEP to descope the runtime configuration (moved it to future work) and removed the implementation details to put focus on the API. Please take a look.

cc @abayer

[jerop]

@afrittoli @pritidesai @vdemeester - please take a look

[lbernick]

@afrittoli @vdemeester @pritidesai let's try to unblock Andrew in his implementation

[vdemeester]

One naive question : are we committed to have this in prior to v1 ?

[tekton-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lbernick, vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• teps/OWNERS [lbernick,vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jerop]

One naive question : are we committed to have this in prior to v1 ?

@vdemeester it's not a blocker for v1, but hoping we'd have it available by then behind the feature flags - made this explicit in #735 - thank you for the review!

[jerop]

Can we merge this offline? It meets the requirement for 2 approvers from different companies and the TEP is still in proposed state - planning to open another PR adding design details and making it as implementable

(cc @abayer @afrittoli @dibyom @pritidesai @vdemeester)

[vdemeester]

/test .*

[dibyom]

Looks like the open questions are around the future work from this TEP, so I think we can merge this

/lgtm