[Testing] KFP standalone test infra for upgradability

[Bobgy]

Fixes #1638
/area testing

---

This change is 

[Bobgy]

Verified locally, the process of deploy -> upgrade works. I need to add some integration tests to verify resources.

[Bobgy]

/retest

[Bobgy]

/retest

[Bobgy]

/assign @IronPan

Not fully finished on upgrade_test.go, but you can first review overall structure of the upgrade test.

[Bobgy]

Note that upgrade tests are passing, even though they don't show here: https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubeflow_pipelines/1971/kubeflow-pipeline-upgrade-test/1192674908237205504

I will update https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubeflow/kubeflow-presubmits.yaml#L337 to show upgrade tests once this PR is merged.

[IronPan]

Mostly LGTM. Thanks Yuan. Could you check why job.trigger is inconsistent

[Bobgy]

/retest
to verify test flakiness

[Bobgy]

@IronPan thanks, I will verify that

[Bobgy]

@IronPan
The difference problem I mentioned before was

Diff:
            	--- Expected
            	+++ Actual
            	@@ -30,3 +30,7 @@
            	   CronSchedule: (*job_model.APICronSchedule)(<nil>),
            	-  PeriodicSchedule: (*job_model.APIPeriodicSchedule)(<nil>)
            	+  PeriodicSchedule: (*job_model.APIPeriodicSchedule)({
            	+   EndTime: (strfmt.DateTime) 0001-01-01T00:00:00.000Z,
            	+   IntervalSecond: (int64) 0,
            	+   StartTime: (strfmt.DateTime) 0001-01-01T00:00:00.000Z
            	+  })
            	  }),
Test:       	TestUpgrade/TestVerify

I think it doesn't affect behavior, might be caused by some inconsistency of writing to and reading from db model.

UPDATE: I found the culprit: https://github.com/Bobgy/pipelines/blob/bec3c94a4505de4ff9129a8e598ef9aa2d6dff17/backend/src/apiserver/storage/job_store.go#L358
When updating a Job from persistence agent, it defaults interval value to 0. This is fine, cron string is also defaulted to empty string.
However, when converting model to api: https://github.com/Bobgy/pipelines/blob/bec3c94a4505de4ff9129a8e598ef9aa2d6dff17/backend/src/apiserver/server/api_converter.go#L305, cron string checks for empty string, but interval doesn't check for 0.

I will fix it in this PR.
This problem isn't triggered in original integration tests, because creating and fetching the job was too fast. Persistence agent didn't get a chance to update Job from swf CR. However, with upgrade tests, there are long enough time for persistence agent to update.

[Bobgy]

/retest

[Bobgy]

/test kubeflow-pipeline-upgrade-test

[Bobgy]

/test kubeflow-pipeline-upgrade-test
strange, error messages says the test was interrupted, I'm guessing that's a prow issue because this job doesn't have enough priority

[IronPan]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: IronPan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• backend/OWNERS [IronPan]
• test/OWNERS [IronPan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Bobgy]

/retest