Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
perf(pipeline): Improve execution times for dense pipeline graphs (#4824
) * test(pipeline): Define StartStageHandler performance when given a complex pipeline with multiple layers of upstream stages * fix(pipeline): Memoize anyUpstreamStagesFailed results This turns the anyUpstreamStagesFailed calculation from one that scales exponentially based on the number of (branches+downstream stages) in a pipeline to one that scales linearly based on the total number of stages in a pipeline. This is a significant performance improvement, especially for very large and complicated pipelines * refactor(stage): Move anyUpstreamStagesFailed to StartStageHandler since that's the only place where it gets used * fix(stage): Avoid using a ConcurrentHashMap for memoization. The recursive anyUpstreamStagesFailed(StageExecution) function runs in a single thread, so ConcurrentHashMap is not necessary here * docs(test): Use a more concise test name * perf(stage): First check if a stage has been visited before checking for parent stages. stage.getRequisiteStageRefIds is a more expensive call because the underlying implementation creates a copy of a List. Therefore, start with the cheaper operation first hoping to short-circuit and avoid the more expensive check * perf(stage): Filter out non-synthetic stages The javadocs state that the syntheticStageOwner property is null for non-synthetic stages. Use this information to filter out non-synthetic stages before attempting a potentially expensive operation to check for synthetic parents of previousStages * perf(stage): Precompute requisiteStageRefIds StageExecutionImpl.getRequisiteStageRefIds() returns a copy of a Set. This is a costly operation that has the potential to get repeated for every unvisited stage. To avoid this, compute the value before entering a loop * perf(stage): Only use withAuth when needed withAuth is only necessary when starting a stage. Since withAuth is very computationally expensive for complex pipelines, only call it when it is absolutely necessary. * perf(stage): Remove duplicate call to withStage StartStageHandler already makes a call to message.withStage at the beginning of the handle() method. Therefore, this call within the catch block is unnecessary * chore(import): Clean up unused imports --------- Co-authored-by: Daniel Zheng <d.zheng@salesforce.com>
- Loading branch information
