Skip to content

Commit

Permalink
Pipeline Data Corruption
Browse files Browse the repository at this point in the history
The unit test `TestDivide_Example` was acting flakey in the CI pipeline which suggested a flaw in the divide and
multiply operations. When running the test, the expected result would be one of the input values or the division
result in failure cases. This implied that results were either received out of order or were being sorted incorrectly.

The pipeline runner does a final sort on the results, so that ruled out the received out of order possibility. On
inspection of the sorting index on each task, every index was the zero value. This resulted in occasional correct
and incorrect sorting, causing the test flake.

To correct the problem, the test was updated such that the expected result has an index of `1`, leaving all
other tasks with a `0` index.
  • Loading branch information
EasterTheBunny committed May 21, 2024
1 parent 322f3b9 commit 2bf2fa2
Showing 1 changed file with 10 additions and 9 deletions.
19 changes: 10 additions & 9 deletions core/services/pipeline/task.divide_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ package pipeline_test
import (
"fmt"
"math"
"reflect"
"testing"

"github.com/pkg/errors"
Expand Down Expand Up @@ -198,19 +199,17 @@ func TestDivideTask_Overflow(t *testing.T) {
}

func TestDivide_Example(t *testing.T) {
testutils.SkipFlakey(t, "BCF-3236")
t.Parallel()

dag := `
ds1 [type=memo value=10000.1234]
ds1 [type=memo value=10000.1234];
ds2 [type=memo value=100];
ds2 [type=memo value=100]
div_by_ds2 [type=divide divisor="$(ds2)"];
multiply [type=multiply times=10000 index=1];
div_by_ds2 [type=divide divisor="$(ds2)"]
ds1 -> div_by_ds2 -> multiply;
multiply [type=multiply times=10000 index=0]
ds1->div_by_ds2->multiply;
`

db := pgtest.NewSqlxDB(t)
Expand All @@ -223,12 +222,14 @@ ds1->div_by_ds2->multiply;

lggr := logger.TestLogger(t)
_, trrs, err := r.ExecuteRun(testutils.Context(t), spec, vars, lggr)
require.NoError(t, err)

require.NoError(t, err)
require.Len(t, trrs, 4)

finalResult := trrs[3]

assert.Nil(t, finalResult.Result.Error)
require.NoError(t, finalResult.Result.Error)
require.Equal(t, reflect.TypeOf(decimal.Decimal{}), reflect.TypeOf(finalResult.Result.Value))

assert.Equal(t, "1000012.34", finalResult.Result.Value.(decimal.Decimal).String())
}

0 comments on commit 2bf2fa2

Please sign in to comment.