Add a developer document on running scheduler benchmarks #3192

bsalamat · 2019-02-01T22:59:58Z

/sig scheduling

bsalamat · 2019-02-01T23:06:11Z

/assign @misterikkit

guineveresaenger · 2019-02-06T20:43:02Z

/lgtm - thank you for writing this up @bsalamat, several folks on my team loved it already

/cc @eduartua for dev guide integration

k8s-ci-robot · 2019-02-06T20:43:06Z

@guineveresaenger: GitHub didn't allow me to request PR reviews from the following users: for, dev.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/lgtm - thank you for writing this up @bsalamat, several folks on my team loved it already

/cc @eduartua for dev guide integration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

guineveresaenger · 2019-02-06T20:43:29Z

Ugh. Every time.
/lgtm

guineveresaenger · 2019-02-06T20:43:54Z

/cc @eduartua

cblecker · 2019-02-06T21:33:26Z

/approve
/hold

Holding for any other comments. Feel free to remove hold when satisfied.

k8s-ci-robot · 2019-02-06T21:33:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bsalamat, cblecker

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~contributors/devel/sig-scheduling/OWNERS~~ [bsalamat,cblecker]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

misterikkit · 2019-02-06T22:41:35Z

contributors/devel/sig-scheduling/scheduler_benchmarking.md

+as the initialization phase of the benchmark. `minPods` specifies how many pods
+must be created and scheduled as the actual part of benchmarking.
+You may add other items to the above array and run the benchmarks again. For 
+example, if you want to measure performance in a 2000 node cluster when scheduling


I don't feel that changing the test cases should be part of this documentation. Or at the very least, it should not be under the heading "running integration benchmarks"

Perhaps what we need is to have a section in this doc for running benchmarks and another section for writing new ones or modifying the test cases. Any literal references to test names, files, or code may get out of sync with this document, so let's think about whether that can be avoided.

I am not sure if I agree. I myself usually run benchmarks with a lot larger number of pods in order to minimize other overheads. At the same time, we haven't been able to add all those configurations in the code because they timeout in our CI/CD.
I agree that literal references to test names and files requires additional effort in keeping them in sync with documents, but at this point we don't have a much better way of doing this without losing usefulness of the doc.

@misterikkit I see your point. From a viewpoint of usefulness, having concrete examples is really helpful so I'd be in favor of leaving this, perhaps with a disclaimer and a link to current test files?

Based on how I see this get used, it means that people who want to test extra large scenarios have to carry their own patches around, and there is no guarantee that those patches match what other contributors are using for test. Maybe these scenarios are unique enough that it's not important.

I am in favor of having documentation saying, "here is how to add a test case with lots of nodes/pods".

misterikkit · 2019-02-06T22:44:38Z

contributors/devel/sig-scheduling/scheduler_benchmarking.md

+directory. 
+
+```sh
+make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=xxx -bench=BenchmarkScheduling"


how about -bench=. ?

Added it and more information about Go benchmarks.

Did you mean to remove this now that you added the other example?

No, not really. I actually wanted to keep this so that people find an example of how particular benchmarks can be specified in CLI.

misterikkit · 2019-02-06T22:45:03Z

contributors/devel/sig-scheduling/scheduler_benchmarking.md

+```
+
+These benchmarks are located in `./test/integration/scheduler_perf/scheduler_bench_test.go`. 
+The function names start with `BenchmarkScheduling`. At the beginning of each


I will probably add benchmarks that don't follow this naming scheme. (when I get around to it.)

If you plan to add scheduling benchmarks, I think it would make sense if they start with BenchmarkScheduling. If they are non-scheduling benchmarks, they should probably be added to other files.

If they are non-scheduling benchmarks, then they should be in a different package. We can discuss if I ever get a PR mailed out. 😛

That's a fair point! 😄

gnufied · 2019-02-11T21:26:38Z

There is also https://github.com/kubernetes/kubernetes/blob/master/test/integration/scheduler_perf/README.md btw. To someone who does not regularly run scheduler benchmarks can either this document and referred README updated to reflect which one is preferable?

bsalamat · 2019-02-14T20:35:02Z

@gnufied Thanks for pointing it out. The instructions in the README file are not detailed enough. Those who contribute code to the scheduler normally want to run benchmarks with more details and configurations. This document can help in those situations.

misterikkit

/lgtm

misterikkit · 2019-02-20T20:14:07Z

contributors/devel/sig-scheduling/scheduler_benchmarking.md

+directory. 
+
+```sh
+make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=xxx -bench=BenchmarkScheduling"


Did you mean to remove this now that you added the other example?

bsalamat · 2019-02-20T23:33:42Z

/hold cancel

k8s-ci-robot added sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 1, 2019

k8s-ci-robot requested review from grodrigues3 and guineveresaenger February 1, 2019 23:00

k8s-ci-robot added area/developer-guide Issues or PRs related to the developer guide cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 1, 2019

k8s-ci-robot assigned misterikkit Feb 1, 2019

k8s-ci-robot requested a review from eduartua February 6, 2019 20:43

k8s-ci-robot assigned guineveresaenger Feb 6, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 6, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 6, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 6, 2019

misterikkit reviewed Feb 6, 2019

View reviewed changes

bsalamat mentioned this pull request Feb 11, 2019

Fix code to handle delayed binding volumes kubernetes/kubernetes#73863

Merged

bsalamat force-pushed the sched_bench branch from 531f8d2 to a2f102d Compare February 14, 2019 22:30

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 14, 2019

Add a developer document on running scheduler benchmarks

101f7e2

bsalamat force-pushed the sched_bench branch from a2f102d to 101f7e2 Compare February 15, 2019 20:59

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 20, 2019

misterikkit reviewed Feb 20, 2019

View reviewed changes

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 20, 2019

k8s-ci-robot merged commit 3d02f91 into kubernetes:master Feb 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a developer document on running scheduler benchmarks #3192

Add a developer document on running scheduler benchmarks #3192

bsalamat commented Feb 1, 2019

bsalamat commented Feb 1, 2019

guineveresaenger commented Feb 6, 2019

k8s-ci-robot commented Feb 6, 2019

guineveresaenger commented Feb 6, 2019

guineveresaenger commented Feb 6, 2019

cblecker commented Feb 6, 2019

k8s-ci-robot commented Feb 6, 2019

misterikkit Feb 6, 2019

misterikkit Feb 6, 2019

bsalamat Feb 7, 2019

guineveresaenger Feb 8, 2019

misterikkit Feb 20, 2019

misterikkit Feb 6, 2019

bsalamat Feb 15, 2019

misterikkit Feb 20, 2019

bsalamat Feb 20, 2019

misterikkit Feb 6, 2019

bsalamat Feb 7, 2019

misterikkit Feb 20, 2019

bsalamat Feb 20, 2019

gnufied commented Feb 11, 2019

bsalamat commented Feb 14, 2019

misterikkit left a comment

misterikkit Feb 20, 2019

bsalamat commented Feb 20, 2019

Add a developer document on running scheduler benchmarks #3192

Add a developer document on running scheduler benchmarks #3192

Conversation

bsalamat commented Feb 1, 2019

bsalamat commented Feb 1, 2019

guineveresaenger commented Feb 6, 2019

k8s-ci-robot commented Feb 6, 2019

guineveresaenger commented Feb 6, 2019

guineveresaenger commented Feb 6, 2019

cblecker commented Feb 6, 2019

k8s-ci-robot commented Feb 6, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gnufied commented Feb 11, 2019

bsalamat commented Feb 14, 2019

misterikkit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsalamat commented Feb 20, 2019