Migrate to new test-infra

[PatrickXYS]

Which issue is resolved by this Pull Request:
Part of kubeflow/testing#861

Description of your changes:
Start from migrating pytorch-operator first

[PatrickXYS]

/test ?

[PatrickXYS]

/test ?

[PatrickXYS]

/test ?

[PatrickXYS]

/test ?

[PatrickXYS]

/test ?

[aws-kf-ci-bot]

@PatrickXYS: The following commands are available to trigger jobs:

• /test kubeflow-pytorch-operator-presubmit

Use /test all to run all jobs.

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

/test ?

[aws-kf-ci-bot]

@PatrickXYS: The following commands are available to trigger jobs:

• /test kubeflow-pytorch-operator-presubmit

Use /test all to run all jobs.

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[Jeffwan]

@PatrickXYS What's the recent change?

[PatrickXYS]

@Jeffwan There's no change in testing itself, only migrate from old test-infra to new test-infra.

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

  Warning  Failed     104s   kubelet            Error: failed to start container "test": Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: setenv: invalid argument: unknown

Seems like it's related to encoded secret opencontainers/runc#1720

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

handle object: patching object from cluster: merging object with existing state: unable to recognize \"/tmp/ksonnet-mergepatch992297811\": no matches for kind \"Workflow\" in version \"argoproj.io/v1alpha1\""

subprocess.CalledProcessError: Command 'cmd: ks-13 apply kubeflow-pytorch-operator-presubmit-e2e-316-b2e6905-7312-e89b -c workflows exited with code 1' returned non-zero exit status 1.

Argo workflow has been generated, and apply failed to work. Checking

[PatrickXYS]

Not sure if this is one-off error, kicking off to see how it goes

I remembered that I saw this error previously, but don't remember what I did before to get rid of it

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

I see, it should be caused by IAM role does not have access to the argo cluster

# kubectl get ns
error: You must be logged in to the server (Unauthorized)

[PatrickXYS]

Succeeded in creating Argo workflow in argo cluster

# ks-13 apply pytorch-test-e2e-316-b2e6905-9e68 -c workflows
INFO Applying workflows kubeflow-test-infra.pytorch-test-e2e-316-b2e6905-9e68
INFO Creating non-existent workflows kubeflow-test-infra.pytorch-test-e2e-316-b2e6905-9e68

[PatrickXYS]

@Jeffwan I forgot to mention that we should help WG create ECR registry in new AWS account in advance

Also found that test-worker image 527798164940.dkr.ecr.us-west-2.amazonaws.com/aws-kubeflow-ci/test-worker:latest is hard-coded into WG's repo.

Need to gather as a list and then migrate

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

Ref: https://argo.kubeflow-testing.com/workflows/kubeflow-test-infra/kubeflow-pytorch-operator-presubmit-e2e-316-6defc74-8208-14a8?tab=workflow&nodeId=kubeflow-pytorch-operator-presubmit-e2e-316-6defc74-8208-14a8-1546001585

Succeeded as expected

[PatrickXYS]

This travis-ci is queued for resources... It might be better to use GitHub actions for simple tests

[PatrickXYS]

/hold

Let's hold for more experiments

[coveralls]

Coverage increased (+0.1%) to 64.23% when pulling 4fa3260 on PatrickXYS:migrate into 282cbee on kubeflow:master.

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

Not sure why the argo_endpoint still point to the old one, checked it locally and it's referring to new one in my local machine.

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

I see, the run_workflow.sh is embedded into image

https://github.com/kubeflow/testing/blob/master/images/Dockerfile.py3.aws#L83-L84

Need to rebuild the test-worker image

[PatrickXYS]

/test kubeflow-pytorch-operator-presubmit

[PatrickXYS]

It works after re-build test-worker image

[PatrickXYS]

/cc @Jeffwan

Should be good to go

[Jeffwan]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Jeffwan, PatrickXYS

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Jeffwan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[PatrickXYS]

/unhold