Add a developer document on running scheduler benchmarks

contributors/devel/sig-scheduling/scheduler_benchmarking.md

@@ -0,0 +1,85 @@
+# Scheduler Benchmarking
+
+Kubernetes scheduler has integration and e2e benchmarks. It is recommended to
+run integration benchmarks at the time of submitting PRs that could potentially
+impact performance of the scheduler and give before and after results with your
+PR submission.
+
+## Running integration benchmarks
+
+To run integration benchmarks use the following command from inside a Kubernetes
+directory. 
+
+```sh
+make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=xxx -bench=."
+```
+
+You can also provide a benchmark name in order to run a specific set of
+benchmarks. Please refer to [Go documentation on benchmarks](https://golang.org/pkg/testing/#hdr-Benchmarks)
+for more information.
+
+```sh
+make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=xxx -bench=BenchmarkScheduling"
+```
+
+These benchmarks are located in `./test/integration/scheduler_perf/scheduler_bench_test.go`. 
+The function names start with `BenchmarkScheduling`. At the beginning of each
+function there are a few lines in the form of:
+
+```go
+	tests := []struct{ nodes, existingPods, minPods int }{
+		{nodes: 100, existingPods: 1000, minPods: 100},
+		{nodes: 1000, existingPods: 1000, minPods: 100},
+		{nodes: 5000, existingPods: 1000, minPods: 1000},
+	}
+```
+
+Each line indicates a test. `nodes` specifies the number of nodes in the test
+cluster. `existingPods` specifies how many pods should be created and scheduled
+as the initialization phase of the benchmark. `minPods` specifies how many pods
+must be created and scheduled as the actual part of benchmarking.
+You may add other items to the above array and run the benchmarks again. For 
+example, if you want to measure performance in a 5000 node cluster when scheduling
+10000 pods, you can add the following to the `tests` array.
+
+```go
+{nodes: 5000, existingPods: 1000, minPods: 10000},
+```
+
+You can also run a particular configuration from the above `tests` by specifying
+it in the `-bench` argument. For example, the following will run only those
+benchmarks with 5000 nodes and 1000 existing pods.
+
+```sh
+make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=xxx -bench=BenchmarkScheduling/5000Nodes/1000Pods"
+```
+
+## Profiling the scheduler
+
+You can get CPU profiling information from the benchmarks by adding a `-cpuprofile`
+to the command above. Example:
+
+```sh
+make test-integration WHAT=./test/integration/scheduler_perf KUBE_TEST_VMODULE="''" KUBE_TEST_ARGS="-run=xxx -bench=BenchmarkScheduling -cpuprofile cpu.out"
+```
+
+After obtaining the CPU profile, you can use `pprof` to view the results. For
+example, you can use the following command to see the output in your browser: 
+
+```sh
+go tool pprof -web cpu.out
+```
+
+You may also want to read more about profiling [here](../sig-scalability/profiling.md).
+
+## End to end performance numbers
+
+The scheduler has end to end real cluster benchmarks that run regularly. These
+benchmarks are available on our [perf dashboard](http://perf-dash.k8s.io/). Once
+on the perf dashboard, you can choose various sizes of clusters to see the
+scheduler's performance. For example, you can choose
+`gce-5000Nodes > Scheduler > SchedulingThroughput` to see the scheduler's
+throughput for a 5000-node cluster.
+
+Note that the API server rate limitation is set to 100 requests per second. As
+a result, the scheduler's throughput is capped at 100 pods/s.