Modified node-kubelet-flaky job and image spec #17687

alejandrox1 · 2020-05-22T13:29:44Z

This patch aims to enable the performance benchmarks in the
node-kubelet-flaky e2e job to run on an instance with enough cpu and
memory to run performance workloads.
It also explicitly specifies the --server-start-timeout flag to tell
the kubelet how long to wait for healthchecks in between restarts.

xref: kubernetes/kubernetes#91263
Requires kubernetes/kubernetes#91363

Signed-off-by: alejandrox1 alarcj137@gmail.com

/sig node
/priority important-longterm
/kind failing-test

alejandrox1 · 2020-05-22T13:42:22Z

/assign @yujuhong @Random-Liu @dchen1107
ptal

jobs/e2e_node/perf-config.yaml

bart0sh · 2020-05-26T18:32:53Z

/lgtm

MHBauer · 2020-05-27T20:55:07Z

jobs/e2e_node/perf-config.yaml

+  cosbeta-resource1:
+    image: cos-beta-81-12871-44-0
+    project: cos-cloud
+    machine: n1-standard-16


can see from kubernetes/kubernetes#65251 that it is supposed to be running on this machine type.

MHBauer

I like the use of a separate image-config split out.

Are there other areas/tests that do performance checking of an individual node?

Original idea all the way back when kubernetes/kubernetes#65249

MHBauer · 2020-05-28T20:42:59Z

config/jobs/kubernetes/sig-node/node-kubelet.yaml

@@ -177,8 +177,8 @@ periodics:
      - --deployment=node
      - --gcp-project-type=node-e2e-project
      - --gcp-zone=us-west1-b
-      - --node-args=--image-config-file=/workspace/test-infra/jobs/e2e_node/image-config.yaml
-      - --node-test-args= --kubelet-flags="--cgroups-per-qos=true --cgroup-root=/"
+      - --node-args=--image-config-file=/workspace/test-infra/jobs/e2e_node/perf-config.yaml


it's better to use a separate config file as well, because I have no idea why we would need to run 3 copies of this on different operating systems.

jobs/e2e_node/perf-config.yaml

alejandrox1 · 2020-05-30T22:57:55Z

Thank you for the reviews @bart0sh @bsdnet @MHBauer
patched the pr based on your comments.

MHBauer

I don't think we ever fully resolved the thoughts from kubernetes/kubernetes#91263 (comment) & #17669 (comment)

I have an action item to write an email to the mailing list and raise the issue elsewhere to see who might want to own these.

MHBauer · 2020-06-01T23:14:05Z

config/jobs/kubernetes/sig-node/node-kubelet.yaml

@@ -177,8 +177,8 @@ periodics:
      - --deployment=node
      - --gcp-project-type=node-e2e-project
      - --gcp-zone=us-west1-b
-      - --node-args=--image-config-file=/workspace/test-infra/jobs/e2e_node/image-config.yaml
-      - --node-test-args= --kubelet-flags="--cgroups-per-qos=true --cgroup-root=/"
+      - --node-args=--image-config-file=/workspace/test-infra/jobs/e2e_node/perf-cos-81-12871-103-0-config.yaml


sorry, I'll be more explicit, I think the filename should be perf-image-config.yaml

haha 😅 fixed it!

MHBauer

This was the flaky job:

➜  e2e_node git:(raise-dockershim-error) rg '\[Flaky\]' -r .
node_perf_test.go
60:var _ = SIGDescribe("Node Performance Testing [Serial] [Slow] .", func() {

It just turns out there's only the one in it right now. So it's a happy accident that these tests are isolated to a single job.

MHBauer · 2020-06-09T18:05:36Z

jobs/e2e_node/perf-image-config.yaml

@@ -0,0 +1,9 @@
+---
+images:


I don't know who monitors the output of this (clearly nobody), nor do I know if it surfaces in a dashboard.

MHBauer · 2020-06-09T18:39:36Z

jobs/e2e_node/perf-image-config.yaml

+    machine: n1-standard-16
+    metadata: "user-data<test/e2e_node/jenkins/gci-init.yaml,gci-update-strategy=update_disabled"
+    tests:
+      - 'Node Performance Testing'


It happens to be the only test marked flaky left in node_e2e.

I think the idea of both Flaky regex and 'Node Performance Testing' is for this to be moved out of flaky and into it's own job eventually. (It should probably be it's own job already, because nothing else is backed by a n1-standard-16.

alejandrox1 · 2020-06-10T00:03:22Z

Thank you everyone for your comments!
I added an ubuntu image to run these tests on.

bsdnet

s/benchamrks/benchmarks/g

bsdnet · 2020-06-10T07:53:21Z

jobs/e2e_node/perf-image-config.yaml

+---
+images:
+  cos-stable1:
+    image: cos-81-12871-103-0


nit: cos-81-12871-119-0

alejandrox1 · 2020-06-10T12:03:05Z

s/benchamrks/benchmarks/g

@bsdnet what do you mean?

bart0sh · 2020-06-10T12:19:49Z

s/benchamrks/benchmarks/g

@bsdnet what do you mean?

that means 'replace benchamrks with benchmarks everywhere', I guess. However, I could only find it in the commit message.

This patch aims to enable the performance benchmarks in the node-kubelet-flaky e2e job to run on an instance with enough cpu and memory to run performance workloads. It also explicitly specifies the `--server-start-timeout` flag to tell the kubelet how long to wait for healthchecks in between restarts. Signed-off-by: alejandrox1 <alarcj137@gmail.com>

alejandrox1 · 2020-06-10T14:22:52Z

haha thank you for the pointer!
Fixed the spelling of my commit.
I was worried I had edited some other file or something.

spiffxp · 2020-06-10T17:28:52Z

/approve
/lgtm

k8s-ci-robot · 2020-06-10T17:29:11Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alejandrox1, bsdnet, MHBauer, spiffxp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~config/jobs/kubernetes/sig-node/OWNERS~~ [spiffxp]
~~jobs/e2e_node/OWNERS~~ [spiffxp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2020-06-10T17:50:58Z

@alejandrox1: Updated the job-config configmap in namespace default at cluster default using the following files:

key node-kubelet.yaml using file config/jobs/kubernetes/sig-node/node-kubelet.yaml

In response to this:

This patch aims to enable the performance benchmarks in the
node-kubelet-flaky e2e job to run on an instance with enough cpu and
memory to run performance workloads.
It also explicitly specifies the --server-start-timeout flag to tell
the kubelet how long to wait for healthchecks in between restarts.

xref: kubernetes/kubernetes#91263
Requires kubernetes/kubernetes#91363

Signed-off-by: alejandrox1 alarcj137@gmail.com

/sig node
/priority important-longterm
/kind failing-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

MHBauer · 2020-06-11T00:38:51Z

looks like tensorflow works okay, integersort works okay, and then embarrassingly parallel workload blows up.

k8s-ci-robot requested review from Random-Liu and yujuhong May 22, 2020 13:30

k8s-ci-robot added area/config Issues or PRs related to code in /config area/jobs sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels May 22, 2020

alejandrox1 mentioned this pull request May 22, 2020

Tuned npb is workload resources kubernetes/kubernetes#91363

Merged

k8s-ci-robot assigned dchen1107, Random-Liu and yujuhong May 22, 2020

bsdnet reviewed May 26, 2020

View reviewed changes

jobs/e2e_node/perf-config.yaml Outdated Show resolved Hide resolved

jobs/e2e_node/perf-config.yaml Outdated Show resolved Hide resolved

k8s-ci-robot assigned bart0sh May 26, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 26, 2020

MHBauer reviewed May 27, 2020

View reviewed changes

MHBauer suggested changes May 28, 2020

View reviewed changes

k8s-ci-robot assigned MHBauer May 28, 2020

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 28, 2020

alejandrox1 force-pushed the fix-kubelet-flaky-job branch from fad68ce to c25a35d Compare May 30, 2020 22:53

alejandrox1 requested a review from MHBauer May 30, 2020 22:56

bsdnet mentioned this pull request Jun 1, 2020

REQUEST: New membership for bsdnet kubernetes/org#1913

Closed

1 task

MHBauer reviewed Jun 1, 2020

View reviewed changes

alejandrox1 force-pushed the fix-kubelet-flaky-job branch 2 times, most recently from 29e7575 to adb018a Compare June 6, 2020 20:03

k8s-ci-robot requested a review from vpickard June 9, 2020 15:30

MHBauer approved these changes Jun 9, 2020

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2020

alejandrox1 force-pushed the fix-kubelet-flaky-job branch from f0982e8 to 63892f5 Compare June 9, 2020 23:45

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2020

alejandrox1 force-pushed the fix-kubelet-flaky-job branch from 63892f5 to 3971dd0 Compare June 9, 2020 23:55

bsdnet approved these changes Jun 10, 2020

View reviewed changes

k8s-ci-robot assigned bsdnet Jun 10, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 10, 2020

alejandrox1 force-pushed the fix-kubelet-flaky-job branch from 3971dd0 to 8b76cec Compare June 10, 2020 12:01

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 10, 2020

alejandrox1 force-pushed the fix-kubelet-flaky-job branch from 8b76cec to 86e15df Compare June 10, 2020 14:22

k8s-ci-robot assigned spiffxp Jun 10, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 10, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 10, 2020

k8s-ci-robot merged commit a8d5af8 into kubernetes:master Jun 10, 2020

k8s-ci-robot added this to the v1.19 milestone Jun 10, 2020

alejandrox1 deleted the fix-kubelet-flaky-job branch June 10, 2020 19:06

MHBauer mentioned this pull request Jun 11, 2020

ci-kubernetes-node-kubelet-flaky has been constantly failing kubernetes/kubernetes#91263

Closed

MHBauer mentioned this pull request Oct 5, 2020

re-enable all node flaky test, move node performance test to new job #19352

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modified node-kubelet-flaky job and image spec #17687

Modified node-kubelet-flaky job and image spec #17687

alejandrox1 commented May 22, 2020 •

edited

Loading

alejandrox1 commented May 22, 2020

bart0sh commented May 26, 2020

MHBauer May 27, 2020

MHBauer left a comment

MHBauer May 28, 2020

alejandrox1 commented May 30, 2020

MHBauer left a comment

MHBauer Jun 1, 2020

alejandrox1 Jun 6, 2020

MHBauer left a comment

MHBauer Jun 9, 2020

MHBauer Jun 9, 2020

alejandrox1 commented Jun 10, 2020

bsdnet left a comment

bsdnet Jun 10, 2020

alejandrox1 Jun 10, 2020

alejandrox1 commented Jun 10, 2020

bart0sh commented Jun 10, 2020

alejandrox1 commented Jun 10, 2020 •

edited

Loading

spiffxp commented Jun 10, 2020

k8s-ci-robot commented Jun 10, 2020

k8s-ci-robot commented Jun 10, 2020

MHBauer commented Jun 11, 2020

		@@ -0,0 +1,9 @@
		---
		images:

Modified node-kubelet-flaky job and image spec #17687

Modified node-kubelet-flaky job and image spec #17687

Conversation

alejandrox1 commented May 22, 2020 • edited Loading

alejandrox1 commented May 22, 2020

bart0sh commented May 26, 2020

Choose a reason for hiding this comment

MHBauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alejandrox1 commented May 30, 2020

MHBauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MHBauer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alejandrox1 commented Jun 10, 2020

bsdnet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alejandrox1 commented Jun 10, 2020

bart0sh commented Jun 10, 2020

alejandrox1 commented Jun 10, 2020 • edited Loading

spiffxp commented Jun 10, 2020

k8s-ci-robot commented Jun 10, 2020

k8s-ci-robot commented Jun 10, 2020

MHBauer commented Jun 11, 2020

alejandrox1 commented May 22, 2020 •

edited

Loading

alejandrox1 commented Jun 10, 2020 •

edited

Loading