-
Notifications
You must be signed in to change notification settings - Fork 193
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[JENKINS-49073] Parallel step should propagate the worst result when not using failFast #325
Conversation
|
||
@Issue("JENKINS-49073") | ||
@Test | ||
public void parallelPropagatesStatus() throws Exception { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added these new test cases, which exercise the new functionality and fail prior to this change. Are there any other use cases I should consider or test cases I should write?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks nice from a quick glance; have not yet done a detailed review of the logic.
src/main/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStep.java
Outdated
Show resolved
Hide resolved
src/main/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStep.java
Outdated
Show resolved
Hide resolved
src/test/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStepTest.java
Show resolved
Hide resolved
@@ -167,6 +167,12 @@ | |||
<version>1.11</version> | |||
<scope>test</scope> | |||
</dependency> | |||
<dependency> | |||
<groupId>org.jenkins-ci.plugins</groupId> | |||
<artifactId>pipeline-build-step</artifactId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unnecessary I think; pending JENKINS-27092, can use non-sandboxed scripts which directly throw new FlowInterruptedException(…)
for tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps this isn't necessary, but I think it makes the test more realistic. I saw that we're already pulling in pipeline-input-step
, so pulling in pipeline-build-step
as well didn't seem like a big price to pay for the benefit of more realistic integration testing. I think I'd rather keep this as-is, but let me know if you feel strongly otherwise.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No strong feeling.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Personally, I'd lean towards the FlowInterruptedException
approach since it would make parallelPropagatesStatusImpl
much easier to read, but I think either approach is fine.
My most recent test run failed due to an issue that doesn't look related to this change:
Restarting to get another build. |
Now all branches of my build failed without even running a single test. I'm just going to give up for today and try to run a build again tomorrow. |
|
I got further today. My test run passed in all environments except in
I can't reproduce these locally and they don't appear to be related to my changes. I'll leave it to the maintainers to comment on any possible next steps here. |
The two are mutually exclusive and an
I would also like it to be the default, but I erred on the conservative side in this change. There is precedent for changing existing behavior (and breaking backward compatibility!) to adopt a more sensible default in jenkinsci/pipeline-build-step-plugin#24 (comment):
Given this past precedent, it seems possible to change the default behavior of the |
@basil Before jenkinsci/pipeline-build-step-plugin#24, was there any normal scenario in which users would care about the difference between |
I think you meant to ask if there was any normal scenario in which users would care about the difference between I took a look at the 116 Pipeline jobs in use at my company, and I didn't find a single instance where the author manually threw a Based on the above, I think we have a few options:
I personally don't think there is a real use case for propagating the first result without setting |
@basil I think option 2 seems like the most obvious behavior. Unfortunately, we have no real way to know how many users might be relying on the existing behavior, but given that users would have to be manually throwing |
Sounds good to me! I've updated the change to propagate the worst result by default when |
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.SUCCESS, Result.NOT_BUILT); | ||
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.SUCCESS, Result.UNSTABLE, Result.NOT_BUILT); | ||
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.SUCCESS, Result.UNSTABLE, Result.FAILURE, Result.NOT_BUILT); | ||
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.NOT_BUILT, Result.NOT_BUILT); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not entirely satisfied with how the code is currently working for these test cases. For example, for three parallel builds whose results are SUCCESS
, UNSTABLE
, and NOT_BUILT
we are currently propagating a status of NOT_BUILT
. I don't consider this ideal; to me, UNSTABLE
is "worse" than NOT_BUILT
. Similarly, for four parallel builds whose results are SUCCESS
, UNSTABLE
, FAILURE
, and NOT_BUILT
we are currently propagating a status of NOT_BUILT
. I don't consider this ideal; to me, FAILURE
is "worse" than NOT_BUILT
. Finally, for two parallel builds whose status is SUCCESS
and NOT_BUILT
we are currently propagating a status of NOT_BUILT
. This doesn't make sense to me. If all the child jobs had a status of NOT_BUILT
, then I could understanding the parent job having a status of NOT_BUILT
, but as long as a single child job has a status of SUCCESS
, I would expect that to be propagated to the parent.
I will think some more about how to tweak the comparator algorithm to handle these cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Another tricky question is how to handle ABORTED
. Between SUCCESS
and ABORTED
, ABORTED
is clearly worse. But what about between FAILURE
/UNSTABLE
and ABORTED
? I would argue that FAILURE
/UNSTABLE
are "worse", because we know the status of the failed/unstable job (whereas we don't know the status of the aborted job, which never ran to completion). Between NOT_BUILT
and ABORTED
I think ABORTED
is worse.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re: What result is worst, I would just use the ordering defined by Result.ordinal
(used to implement Result.isWorseThan
and friends). Whether that ordering makes the most sense could be debated, but that's the ordering that everything else in Jenkins uses, so I would use it here as well.
Might be interesting to run pipeline-model-definition's tests with this change to see if any of them start failing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Re: What result is worst, I would just use the ordering defined by
Result.ordinal
(used to implementResult.isWorseThan
and friends). Whether that ordering makes the most sense could be debated, but that's the ordering that everything else in Jenkins uses, so I would use it here as well.
My comparison logic is already using Result#isWorseThan
and Result#isBetterThan
, so if we're OK with the behavior demonstrated by the new tests (which I think is acceptable, but not ideal) then I can leave the comparison logic as-is.
Might be interesting to run pipeline-model-definition's tests with this change to see if any of them start failing.
Sure. I opened jenkinsci/pipeline-model-definition-plugin#360 to cover this testing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I opened jenkinsci/pipeline-model-definition-plugin#360 to cover this testing.
These tests passed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To elaborate on my earlier comment, the values in Result
have always seemed weird to me. ABORTED
and NOT_BUILT
feel like a different kind of thing entirely than SUCCESS
, FAILURE
, and UNSTABLE
. Sometimes I think it would have made more sense for results to have 2 parts, a "completion" result, like , NOT_BUILT
, ABORTED
, COMPLETED
, and a "goodness" result, like SUCCESS
, WARNING
, ERROR
. Then you could have something like a Pipeline step that is ABORTED
and SUCCESS
in the case of a user cancelling the build, or ABORTED
and WARNING
in the case of something like a timeout, NOT_BUILT
and SUCCESS
could be used in cases like a Declarative when
expression that is false, and here in the parallel step, the completion result would always be COMPLETED
, and only the "goodness" result would be determined by the results of the branches. Using two values creates other problems though, and not all combinations make sense.
So all that to say, I agree that the behavior here with NOT_BUILT
feels wrong, but I'm not sure if it makes sense to try to change it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with your observations here. The awkwardness in the use cases I described above is exposing a deeper problem in how the Result
API is defined in Jenkins core. When I get a chance I'll try to file a bug/RFE against Jenkins core with the results of this discussion. Perhaps one day that can be addressed, but I agree that it is quite orthogonal to this effort.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks ok to me in general, added some minor comments.
src/test/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStepTest.java
Outdated
Show resolved
Hide resolved
src/main/java/org/jenkinsci/plugins/workflow/cps/steps/ParallelStep.java
Show resolved
Hide resolved
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.SUCCESS, Result.NOT_BUILT); | ||
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.SUCCESS, Result.UNSTABLE, Result.NOT_BUILT); | ||
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.SUCCESS, Result.UNSTABLE, Result.FAILURE, Result.NOT_BUILT); | ||
parallelPropagatesStatusImpl( | ||
Result.NOT_BUILT, Result.NOT_BUILT, Result.NOT_BUILT); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To elaborate on my earlier comment, the values in Result
have always seemed weird to me. ABORTED
and NOT_BUILT
feel like a different kind of thing entirely than SUCCESS
, FAILURE
, and UNSTABLE
. Sometimes I think it would have made more sense for results to have 2 parts, a "completion" result, like , NOT_BUILT
, ABORTED
, COMPLETED
, and a "goodness" result, like SUCCESS
, WARNING
, ERROR
. Then you could have something like a Pipeline step that is ABORTED
and SUCCESS
in the case of a user cancelling the build, or ABORTED
and WARNING
in the case of something like a timeout, NOT_BUILT
and SUCCESS
could be used in cases like a Declarative when
expression that is false, and here in the parallel step, the completion result would always be COMPLETED
, and only the "goodness" result would be determined by the results of the branches. Using two values creates other problems though, and not all combinations make sense.
So all that to say, I agree that the behavior here with NOT_BUILT
feels wrong, but I'm not sure if it makes sense to try to change it.
|
||
private FreeStyleProject getDownstreamJob(Result result) throws IOException { | ||
FreeStyleProject ds = story.j.createFreeStyleProject(); | ||
ds.getBuildersList().add(new TestBuilder() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Maybe simpler to use a Pipeline job for the downstream builds instead:
WorkflowJob ds = story.j.createProject(WorkflowJob.class);
ds.setDefinition(new CpsFlowDefinition("currentBuild.result = " + result, true));
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it makes a big difference one way or the other. I did it this way simply because I was copying an existing example from the pipeline-build-step
plugin's tests.
@@ -167,6 +167,12 @@ | |||
<version>1.11</version> | |||
<scope>test</scope> | |||
</dependency> | |||
<dependency> | |||
<groupId>org.jenkins-ci.plugins</groupId> | |||
<artifactId>pipeline-build-step</artifactId> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Personally, I'd lean towards the FlowInterruptedException
approach since it would make parallelPropagatesStatusImpl
much easier to read, but I think either approach is fine.
Co-Authored-By: Devin Nusbaum <dwnusbaum@users.noreply.github.com>
Improve review suggestions
Thanks for helping me take this across the finish line! At this point I think the only remaining change we need to make is to release |
Co-Authored-By: Devin Nusbaum <dwnusbaum@users.noreply.github.com>
Problem
See JENKINS-49073 (comment).
Solution
Inspired by jenkinsci/pipeline-build-step-plugin#24 (comment). Add a new
propagateWorst
property to theparallel
step that implements the desired behavior. When this is in effect, the "worst" branch propagates its build status, as defined by the ordering Jesse suggested.Implementation
To make this change easier to implement, I first did two commits that simply refactor the existing code. These should be reviewed first to ensure that I didn't break anything.
originalFailure
from aSimpleEntry<String,Throwable>
to just aThrowable
. The key was never used, so this is just a simplification.LinkedHashSet
of failures in insertion order. This allows us to delay choosing the top-level exception until later in the execution incheckAllDone
, which is needed in order to implement the newpropagateWorst
functionality.