Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prow presubmit job to build mlkube.io container. #4951

Merged
merged 22 commits into from
Oct 13, 2017
Merged

Prow presubmit job to build mlkube.io container. #4951

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
cf8ce96
Prow presubmit job to build mlkube.io container.
jlewi Oct 10, 2017
01d41a2
Fix starter.yaml
jlewi Oct 10, 2017
49da9b2
* mlkube image moved to mlkube.io repo.
jlewi Oct 12, 2017
6c44b46
Merge branch 'master' into mlkube_prow
jlewi Oct 12, 2017
39f7008
Restore the mlkube periodic job.
jlewi Oct 12, 2017
d212e7b
Sort the configs.
jlewi Oct 12, 2017
d8ba6c9
Add a context to the presubmit job.
jlewi Oct 12, 2017
b0d10e5
Fix the //prow/config:go_default_test
jlewi Oct 12, 2017
6a4bcec
Sort the configs.
jlewi Oct 12, 2017
7899047
Fix the test.
jlewi Oct 12, 2017
98dfc26
* We don't need jobs/config.json entries; this is legacy Jenkins stuff.
jlewi Oct 13, 2017
8563b0c
Add a test-grid entry.
jlewi Oct 13, 2017
4309c6d
* Don't mount the docker socket or run in privileged mode.
jlewi Oct 13, 2017
7507edb
* Try to fix failure
jlewi Oct 13, 2017
988f657
Revert "* Try to fix failure"
jlewi Oct 13, 2017
5d6a304
Include presubmits from jlewi/mlkube.io
jlewi Oct 13, 2017
0090983
gofmt.
jlewi Oct 13, 2017
80da1c6
Merge branch 'master' into mlkube_prow
jlewi Oct 13, 2017
23987bf
gofmt.
jlewi Oct 13, 2017
9a5f92c
Make the periodic job added by @foxish consistent with the pre/post s…
jlewi Oct 13, 2017
9e76d5f
Address code review comments.
jlewi Oct 13, 2017
6603279
* Remove reference to socket.
jlewi Oct 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 0 additions & 21 deletions jobs/config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8068,27 +8068,6 @@
"sig-cluster-lifecycle"
]
},
"ci-kubernetes-e2e-mlkube-gke": {
"args": [
"--charts",
"--check-leaked-resources",
"--deployment=gke",
"--extract=gke",
"--gcp-cloud-sdk=gs://cloud-sdk-testing/ci/staging",
"--gcp-node-image=gci",
"--gcp-project=mlkube-testing",
"--gcp-zone=us-central1-b",
"--gke-environment=prod",
"--mount-paths=$GOPATH/src/github.com/foxish/mlkube.io:/src/k8s.io/charts",
"--provider=gke",
"--test=false",
"--timeout=50m"
],
"scenario": "kubernetes_e2e",
"sigOwners": [
"sig-big-data"
]
},
"ci-kubernetes-e2e-node-canary": {
"args": [
"--deployment=node",
Expand Down
7 changes: 7 additions & 0 deletions jobs/config_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -789,6 +789,13 @@ def test_all_project_are_unique(self):
'ci-kubernetes-node-kubelet-conformance': 'ci-kubernetes-node-kubelet-*',
'ci-kubernetes-node-kubelet-benchmark': 'ci-kubernetes-node-kubelet-*',
'ci-kubernetes-node-kubelet': 'ci-kubernetes-node-kubelet-*',

# The mlkube projects intentionally share projects,
# We map each test name to the same value so that below when we check for number of tests
# using this project there will just be one entry.
'mlkube-build-presubmit': 'mlkube-*',
'mlkube-build-postsubmit': 'mlkube-*',
'ci-kubernetes-e2e-mlkube-gke': 'mlkube-*',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now you probably don't need this block. The test basically scans duplicated projects from config.json.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

}
for soak_prefix in [
'ci-kubernetes-soak-gce-1.5',
Expand Down
75 changes: 54 additions & 21 deletions prow/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2489,6 +2489,31 @@ presubmits:
hostPath:
path: /mnt/disks/ssd0

jlewi/mlkube.io:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hummm, I feel like our bot might not have access to your repo, you might want to configure somewhere around https://github.com/kubernetes/test-infra/blob/master/prow/plugins.yaml#L5-L11

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also https://github.com/kubernetes/test-infra/blob/master/prow/plugins.yaml#L37-L38

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I also think I need to configure webhooks on my repo but I'll do that after this PR is done.

- name: mlkube-build-presubmit
context: mlkube-build-presubmit
agent: kubernetes
always_run: true # Run for every PR, or only when requested.
rerun_command: "/test mlkube-build-presubmit"
trigger: "(?m)^/test( all| mlkube-build-presubmit),?(\\s+|$)"
branches:
- master
spec:
containers:
# TODO(jlewi): Replace latest with a specific tag once the images stabilize.
- image: gcr.io/mlkube-testing/builder:latest
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/service-account/service-account.json
volumeMounts:
- name: service
mountPath: /etc/service-account
readOnly: true
volumes:
- name: service
secret:
secretName: service-account

postsubmits:
kubernetes/kubernetes:
- name: ci-kubernetes-bazel-build
Expand Down Expand Up @@ -3263,6 +3288,29 @@ postsubmits:
secret:
secretName: service-account

jlewi/mlkube.io:
- name: mlkube-build-postsubmit
agent: kubernetes
branches:
- master
spec:
containers:
# TODO(jlewi): Replace latest with a specific tag once the images stabilize.
- image: gcr.io/mlkube-testing/builder:latest
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/service-account/service-account.json
volumeMounts:
- name: service
mountPath: /etc/service-account
readOnly: true
volumes:
- name: service
secret:
secretName: service-account
- name: docker
hostPath:
path: /var/run/docker.sock
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove the sock map here as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


periodics:
- interval: 1h
Expand Down Expand Up @@ -15700,39 +15748,24 @@ periodics:
defaultMode: 256
secretName: aws-cred

- interval: 30m
- interval: 8h
agent: kubernetes
name: ci-kubernetes-e2e-mlkube-gke
name: mlkube-build-periodic
spec:
containers:
- args:
- --timeout=70
- --repo=github.com/foxish/mlkube.io
# TODO(jlewi): Replace latest with a specific tag once the images stabilize.
- image: gcr.io/mlkube-testing/builder:latest
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /etc/service-account/service-account.json
- name: USER
value: prow
- name: JENKINS_GCE_SSH_PRIVATE_KEY_FILE
value: /etc/ssh-key-secret/ssh-private
- name: JENKINS_GCE_SSH_PUBLIC_KEY_FILE
value: /etc/ssh-key-secret/ssh-public
image: gcr.io/k8s-testimages/kubekins-e2e:v20171011-8265ed12-master
volumeMounts:
- mountPath: /etc/service-account
name: service
readOnly: true
- mountPath: /etc/ssh-key-secret
name: ssh
- name: service
mountPath: /etc/service-account
readOnly: true
volumes:
- name: service
secret:
secretName: service-account
- name: ssh
secret:
defaultMode: 256
secretName: ssh-key-secret

- name: ci-kubernetes-e2e-node-canary
interval: 1h
Expand Down
2 changes: 0 additions & 2 deletions prow/config/jobs_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,6 @@ func TestPresubmits(t *testing.T) {
job.Name = strings.Replace(job.Name, "pull-security-kubernetes", "pull-kubernetes", 1)
if j, present := bootstrapConfig[job.Name]; present {
scenario = fmt.Sprintf("scenarios/%s.py", j.Scenario)
} else {
scenario = fmt.Sprintf("jobs/%s.sh", job.Name)
}

// Ensure that jobs have a shell script of the same name.
Expand Down
22 changes: 15 additions & 7 deletions testgrid/config/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1535,8 +1535,12 @@ test_groups:
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-master-gci-new-downgrade-cluster-parallel
- name: ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-gci-new-gci-master-upgrade-cluster-new
- name: ci-kubernetes-e2e-mlkube-gke
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-e2e-mlkube-gke
- name: mlkube-build-periodic
gcs_prefix: kubernetes-jenkins/logs/mlkube-build-periodic
- name: mlkube-build-presubmit
gcs_prefix: kubernetes-jenkins/logs/mlkube-build-presubmit
- name: mlkube-build-postsubmit
gcs_prefix: kubernetes-jenkins/logs/mlkube-build-postsubmit
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also add to https://github.com/jlewi/test-infra/blob/8563b0c1fa76dfc890d05f38e6b9bb2faa5f76e6/testgrid/config/config.yaml#L3087-L3092

# kube-proxy daemonset migration jobs
- name: ci-kubernetes-e2e-gci-gce-latest-downgrade-kube-proxy-ds
gcs_prefix: kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce-latest-downgrade-kube-proxy-ds
Expand Down Expand Up @@ -3081,8 +3085,6 @@ dashboards:
dashboard_tab:
- name: charts-gce
test_group_name: ci-kubernetes-charts-gce
- name: mlkube-gke
test_group_name: ci-kubernetes-e2e-mlkube-gke

- name: perf-tests
dashboard_tab:
Expand Down Expand Up @@ -3540,9 +3542,15 @@ dashboards:

- name: sig-big-data
dashboard_tab:
- name: mlkube-gke
description: what does this tab do exactly
test_group_name: ci-kubernetes-e2e-mlkube-gke
- name: mlkube-build-periodic
description: Periodic builds and testing of TfJob CRD at head running on stable GKE.
test_group_name: mlkube-build-periodic
- name: mlkube-build-postsubmit
description: Postsubmit tests of TfJob CRD running on stable GKE.
test_group_name: mlkube-build-postsubmit
- name: mlkube-build-presubmit
description: Presubmit tests of TfJob CRD running on stable GKE.
test_group_name: mlkube-build-presubmit
notifications:
- summary: Please configure this skeleton dashboard and remove this notification
context_link: https://github.com/kubernetes/test-infra/tree/master/testgrid/config
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(probably can delete this notification now)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Expand Down
2 changes: 1 addition & 1 deletion testgrid/jenkins_verify/jenkins_validate.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ func main() {
}

// Also check k/k presubmit, prow postsubmit and periodic jobs
for _, job := range prowConfig.AllPresubmits([]string{"kubernetes/kubernetes"}) {
for _, job := range prowConfig.AllPresubmits([]string{"jlewi/mlkube.io", "kubernetes/kubernetes"}) {
jobs[job.Name] = false
}

Expand Down