Our test code and test images code is not always the same.

[Ark-kun]

This issue manifested itself several times. The latest was today when the prow tests were fixed.

Out test images only use the branch code.

git clone https://github.com/kubeflow/pipelines /ml
git checkout 6e96b054fb2585f3577155fa92dd107c6e1b5dd2

But the tests that Prow runs are taken from the result of merging the base branch (master) with the PR branch.

I1103 23:36:06.094] Checkout: /workspace/github.com/googleprivate/ml master:296b540cd724fed645e1652f12428462fd5375ed,1532:5afe507591f58f76a12c9f0f3b6659a30b657060 to /workspace/github.com/googleprivate/ml
I1103 23:36:06.094] Call:  git init github.com/googleprivate/ml
I1103 23:36:06.101] Call:  git clean -dfx
I1103 23:36:06.105] Call:  git reset --hard
I1103 23:36:06.110] Call:  git config --local user.name 'K8S Bootstrap'
I1103 23:36:06.116] Call:  git config --local user.email k8s_bootstrap@localhost
I1103 23:36:06.122] Call:  git fetch --quiet --tags git@github.com:googleprivate/ml master +refs/pull/1532/head:refs/pr/1532
I1103 23:36:10.765] Call:  git checkout -B test 296b540cd724fed645e1652f12428462fd5375ed
I1103 23:36:11.199] Call:  git show -s --format=format:%ct HEAD
I1103 23:36:11.204] Call:  git merge --no-ff -m 'Merge +refs/pull/1532/head:refs/pr/1532' 5afe507591f58f76a12c9f0f3b6659a30b657060

This effectively means that the test code is always taken from master while test image code is taken from the branch and may be out of sync.

We should also do something like

git clone https://github.com/kubeflow/pipelines
cd pipelines
git merge --no-ff 321ca814db4955b3950b0fac06a2d289fe4db39a -m "Merged PR"