kvm2 driver: Add flag --kvm-numa-count" support topology-manager simulate numa #10471

phantooom · 2021-02-13T15:51:21Z

kvm2 driver support simulate numa node
ref: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/#topology-manager-policies

you can use --kvm-numa-count define kvm vm numa count, use --extra-config=kubelet.topology-manager-policy=none/best-effort
/restricted/single-numa-node config kubelet Topology Manager Policies

minikube start --driver=kvm2  --kvm-numa-count=2 --extra-config=kubelet.topology-manager-policy=best-effort

❯ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ /bin/toolbox 
Trying to pull docker://fedora:latest...
Getting image source signatures
Copying blob f147208a1e03 done  
Copying config a78267678b done  
Writing manifest to image destination
Storing signatures
8818f2a63b341c3bcfa41fc6879115e692482d0d65c450029cc9bdcba5879c2d
8818f2a63b341c3bcfa41fc6879115e692482d0d65c450029cc9bdcba5879c2d
Untagged: docker.io/library/fedora:latest
Deleted: a78267678b7e6e849c7e960b09227b737a38d5073a5071b041a16bd4b609ef92
Spawning container docker-fedora-latest on /var/lib/toolbox/docker-fedora-latest.
Press ^] three times within 1s to kill container.
[root@minikube ~]# yum install numactl -y
[root@minikube ~]# numactl  -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 0 size: 16034 MB
node 0 free: 15349 MB
node 1 cpus: 4 5 6 7
node 1 size: 16126 MB
node 1 free: 13300 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

closes #10668

k8s-ci-robot · 2021-02-13T15:51:29Z

Welcome @phantooom!

It looks like this is your first PR to kubernetes/minikube 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/minikube has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-02-13T15:51:29Z

Hi @phantooom. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

minikube-bot · 2021-02-13T15:52:04Z

Can one of the admins verify this patch?

medyagh · 2021-02-17T19:27:35Z

@prezha do u approve this PR ?

pkg/drivers/kvm/numa.go

ilya-zuyev · 2021-02-17T19:34:16Z

/ok-to-test

minikube-pr-bot · 2021-02-17T19:42:28Z

kvm2 Driver
error collecting results for kvm2 driver: timing run 0 with Minikube (PR 10471): timing cmd: [/home/performance-monitor/.minikube/minikube-binaries/10471/minikube start --driver=kvm2]: starting cmd: fork/exec /home/performance-monitor/.minikube/minikube-binaries/10471/minikube: exec format error
docker Driver
error collecting results for docker driver: timing run 0 with Minikube (PR 10471): timing cmd: [/home/performance-monitor/.minikube/minikube-binaries/10471/minikube start --driver=docker]: starting cmd: fork/exec /home/performance-monitor/.minikube/minikube-binaries/10471/minikube: exec format error

prezha · 2021-02-18T02:03:12Z

@phantooom , thank you for your pr

the doc you've referenced mentions minikube as means to try control topology management on a node, so generally it would make sense to support it

check requirements:

the same doc states that supported configurations are:

on k8s v1.18 and later - user can specify any version (via '--kubernetesVersion' flag)
up to 8 numa nodes
(as you've commented in code) with '--driver=kvm[2]'
check these requirements accordingly and error in case of a mismatch

examples:

can you amend pr description with examples of simulated numa nodes utilised in minikube with different scopes and policies (you can use '--extra-config' flag for passing respective params to kubelet)

phantooom · 2021-02-18T16:02:30Z

@phantooom , thank you for your pr

the doc you've referenced mentions minikube as means to try control topology management on a node, so generally it would make sense to support it

check requirements:

the same doc states that supported configurations are:

on k8s v1.18 and later - user can specify any version (via '--kubernetesVersion' flag)

up to 8 numa nodes

(as you've commented in code) with '--driver=kvm[2]'
check these requirements accordingly and error in case of a mismatch

examples:

can you amend pr description with examples of simulated numa nodes utilised in minikube with different scopes and policies (you can use '--extra-config' flag for passing respective params to kubelet)

i add some check requirements @prezha

./out/minikube start --driver=kvm2  --kvm-numa-count=2 --extra-config=kubelet.topology-manager-policy=single-numa-node --kubernetes-version=1.13.0
😄  minikube v1.17.1 on Arch 20.2.1
    ▪ KUBECONFIG=/home/phantooom/.kube/all.yml
✨  Using the kvm2 driver based on user configuration

❌  Exiting due to MK_USAGE: numa node is only supported on k8s v1.18 and later


❯ ./out/minikube start --driver=kvm2  --kvm-numa-count=20
😄  minikube v1.17.1 on Arch 20.2.1
    ▪ KUBECONFIG=/home/phantooom/.kube/all.yml
✨  Using the kvm2 driver based on user configuration

❌  Exiting due to MK_USAGE: --kvm-numa-count range is 1-8

minikube-pr-bot · 2021-02-18T16:20:26Z

kvm2 Driver
Times for minikube: 70.5s 68.2s 71.1s
Average time for minikube: 69.9s

Times for Minikube (PR 10471): 70.6s 69.7s 69.0s
Average time for Minikube (PR 10471): 69.8s

Averages Time Per Log

+--------------------------------------------+----------+---------------------+
|                    LOG                     | MINIKUBE | MINIKUBE (PR 10471) |
+--------------------------------------------+----------+---------------------+
| * minikube v1.17.1 on Debian               | 0.0s     | 0.0s                |
| 9.11 (kvm/amd64)                           |          |                     |
| * Using the kvm2 driver based              | 0.0s     | 0.0s                |
| on user configuration                      |          |                     |
| * Starting control plane node              | 0.0s     | 0.0s                |
| minikube in cluster minikube               |          |                     |
| * Creating kvm2 VM (CPUs=2,                | 43.2s    | 43.0s               |
| Memory=3700MB, Disk=20000MB)               |          |                     |
| ...                                        |          |                     |
| * Preparing Kubernetes v1.20.2             | 16.5s    | 16.8s               |
| on Docker 20.10.2 ...                      |          |                     |
|   - Generating certificates                | 1.3s     | 1.4s                |
| and keys ...                               |          |                     |
|   - Booting up control plane               | 5.2s     | 5.4s                |
| ...                                        |          |                     |
|   - Configuring RBAC rules ...             | 1.0s     | 1.1s                |
| * Verifying Kubernetes                     | 0.1s     | 0.1s                |
| components...                              |          |                     |
|   - Using image                            | 1.6s     | 1.6s                |
| gcr.io/k8s-minikube/storage-provisioner:v4 |          |                     |
| * Enabled addons:                          | 0.9s     | 0.3s                |
| storage-provisioner,                       |          |                     |
| default-storageclass                       |          |                     |
| * Done! kubectl is now                     | 0.0s     | 0.0s                |
| configured to use "minikube"               |          |                     |
| cluster and "default"                      |          |                     |
| namespace by default                       |          |                     |
+--------------------------------------------+----------+---------------------+

docker Driver
Times for minikube: 28.2s 28.2s 28.3s
Average time for minikube: 28.3s

Times for Minikube (PR 10471): 28.7s 29.4s 27.3s
Average time for Minikube (PR 10471): 28.5s

Averages Time Per Log

+--------------------------------------------+----------+---------------------+
|                    LOG                     | MINIKUBE | MINIKUBE (PR 10471) |
+--------------------------------------------+----------+---------------------+
| * minikube v1.17.1 on Debian               | 0.2s     | 0.2s                |
| 9.11 (kvm/amd64)                           |          |                     |
| * Using the docker driver                  | 0.1s     | 0.1s                |
| based on user configuration                |          |                     |
| * Starting control plane node              | 0.1s     | 0.1s                |
| minikube in cluster minikube               |          |                     |
| * Creating docker container                | 10.9s    | 11.2s               |
| (CPUs=2, Memory=3700MB) ...                |          |                     |
| * Preparing Kubernetes v1.20.2             | 15.9s    | 15.7s               |
| on Docker 20.10.2 ...                      |          |                     |
| * Verifying Kubernetes                     | 0.1s     | 0.1s                |
| components...                              |          |                     |
|   - Using image                            | 0.9s     | 1.0s                |
| gcr.io/k8s-minikube/storage-provisioner:v4 |          |                     |
| * Enabled addons:                          | 0.1s     | 0.1s                |
| storage-provisioner,                       |          |                     |
| default-storageclass                       |          |                     |
| * Done! kubectl is now                     | 0.0s     | 0.0s                |
| configured to use "minikube"               |          |                     |
| cluster and "default"                      |          |                     |
| namespace by default                       |          |                     |
+--------------------------------------------+----------+---------------------+

medyagh · 2021-02-20T00:27:23Z

@phantooom do u mind running make lint and make test
the lint and uint test are failing

phantooom · 2021-02-20T16:29:17Z

fixed @medyagh

priyawadhwa

Thanks for opening this PR! I left a comment.

cmd/minikube/cmd/start_flags.go

prezha · 2021-02-26T19:36:02Z

@priyawadhwa i think a value of 1 is the default (and minimum) number of nodes in numa arch (ie, all the resources are placed in that single node):

❯ lscpu

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
...
NUMA node(s):                    1
...

pkg/drivers/kvm/numa_test.go

pkg/drivers/kvm/numa.go

medyagh · 2021-03-01T17:56:03Z

plz make sure to run make lint

minikube-pr-bot · 2021-03-02T19:31:12Z

kvm2 Driver
Times for minikube: 70.6s 71.0s 70.9s
Average time for minikube: 70.8s

Times for Minikube (PR 10471): 69.2s 67.9s 68.4s
Average time for Minikube (PR 10471): 68.5s

Averages Time Per Log

+--------------------------------------------+----------+---------------------+
|                    LOG                     | MINIKUBE | MINIKUBE (PR 10471) |
+--------------------------------------------+----------+---------------------+
| * minikube v1.18.0 on Debian               | 0.0s     | 0.0s                |
| 9.11 (kvm/amd64)                           |          |                     |
| * Using the kvm2 driver based              | 0.0s     | 0.0s                |
| on user configuration                      |          |                     |
| * Starting control plane node              | 0.0s     | 0.0s                |
| minikube in cluster minikube               |          |                     |
| * Creating kvm2 VM (CPUs=2,                | 43.7s    | 42.3s               |
| Memory=3700MB, Disk=20000MB)               |          |                     |
| ...                                        |          |                     |
| * Preparing Kubernetes v1.20.2             | 23.9s    | 9.6s                |
| on Docker 20.10.3 ...                      |          |                     |
|   - Generating certificates                | 0.0s     | 2.3s                |
| and keys ...                               |          |                     |
|   - Booting up control plane               | 0.0s     | 10.4s               |
| ...                                        |          |                     |
|   - Configuring RBAC rules ...             | 0.6s     | 1.3s                |
| * Verifying Kubernetes                     | 0.1s     | 0.1s                |
| components...                              |          |                     |
|   - Using image                            | 1.9s     | 1.6s                |
| gcr.io/k8s-minikube/storage-provisioner:v4 |          |                     |
| * Enabled addons:                          | 0.7s     | 1.0s                |
| storage-provisioner,                       |          |                     |
| default-storageclass                       |          |                     |
| * Done! kubectl is now                     | 0.0s     | 0.0s                |
| configured to use "minikube"               |          |                     |
| cluster and "default"                      |          |                     |
| namespace by default                       |          |                     |
+--------------------------------------------+----------+---------------------+

docker Driver
Times for minikube: 27.1s 27.8s 35.8s
Average time for minikube: 30.3s

Times for Minikube (PR 10471): 25.9s 30.6s 26.0s
Average time for Minikube (PR 10471): 27.5s

Averages Time Per Log

+--------------------------------------------+----------+---------------------+
|                    LOG                     | MINIKUBE | MINIKUBE (PR 10471) |
+--------------------------------------------+----------+---------------------+
| * minikube v1.18.0 on Debian               | 0.2s     | 0.2s                |
| 9.11 (kvm/amd64)                           |          |                     |
| * Using the docker driver                  | 0.1s     | 0.1s                |
| based on user configuration                |          |                     |
| * Starting control plane node              | 1.6s     | 0.1s                |
| minikube in cluster minikube               |          |                     |
| * Creating docker container                | 11.8s    | 11.0s               |
| (CPUs=2, Memory=3700MB) ...                |          |                     |
| * Preparing Kubernetes v1.20.2             | 15.4s    | 15.1s               |
| on Docker 20.10.3 ...                      |          |                     |
| * Verifying Kubernetes                     | 0.1s     | 0.1s                |
| components...                              |          |                     |
|   - Using image                            | 1.0s     | 0.9s                |
| gcr.io/k8s-minikube/storage-provisioner:v4 |          |                     |
| * Enabled addons:                          | 0.1s     | 0.1s                |
| storage-provisioner,                       |          |                     |
| default-storageclass                       |          |                     |
| * Done! kubectl is now                     | 0.0s     | 0.0s                |
| configured to use "minikube"               |          |                     |
| cluster and "default"                      |          |                     |
| namespace by default                       |          |                     |
+--------------------------------------------+----------+---------------------+

k8s-ci-robot · 2021-03-03T00:02:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: medyagh, phantooom

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [medyagh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 13, 2021

k8s-ci-robot requested review from afbjorklund and sharifelgamal February 13, 2021 15:51

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 13, 2021

phantooom force-pushed the master branch 2 times, most recently from ad961ad to 1963141 Compare February 13, 2021 16:25

ilya-zuyev reviewed Feb 17, 2021

View reviewed changes

pkg/drivers/kvm/numa.go Outdated Show resolved Hide resolved

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 17, 2021

prezha mentioned this pull request Feb 19, 2021

REQUEST: New membership for prezha kubernetes/org#2514

Closed

6 tasks

phantooom force-pushed the master branch from b390668 to 898057e Compare February 22, 2021 14:23

priyawadhwa suggested changes Feb 26, 2021

View reviewed changes

cmd/minikube/cmd/start_flags.go Show resolved Hide resolved

phantooom requested review from priyawadhwa and ilya-zuyev February 27, 2021 08:52

medyagh requested changes Mar 1, 2021

View reviewed changes

pkg/drivers/kvm/numa_test.go Outdated Show resolved Hide resolved

pkg/drivers/kvm/numa.go Outdated Show resolved Hide resolved

pkg/drivers/kvm/numa.go Outdated Show resolved Hide resolved

medyagh added this to the v.1.19.0-candidate milestone Mar 1, 2021

phantooom added 3 commits March 2, 2021 22:27

kvm2 driver support simulate numa node

e11f23a

add more check for kvm2 support simulate numa node

badc5d4

fix: fix kvm2 numa simulate ut&lint

529881e

phantooom force-pushed the master branch from 898057e to 529881e Compare March 2, 2021 14:40

phantooom requested a review from medyagh March 2, 2021 14:41

medyagh approved these changes Mar 3, 2021

View reviewed changes

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 3, 2021

medyagh changed the title ~~kvm2 driver support simulate numa node~~ kvm2 driver support for topology-manager simulate numa node. Mar 3, 2021

medyagh changed the title ~~kvm2 driver support for topology-manager simulate numa node.~~ kvm2 driver: Add flag --kvm-numa-count" support topology-manager simulate numa Mar 3, 2021

medyagh merged commit 5da07f1 into kubernetes:master Mar 3, 2021

fmotrifork mentioned this pull request Mar 5, 2021

minikube 1.18.1 fishworks/fish-food#1274

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvm2 driver: Add flag --kvm-numa-count" support topology-manager simulate numa #10471

kvm2 driver: Add flag --kvm-numa-count" support topology-manager simulate numa #10471

phantooom commented Feb 13, 2021 •

edited by medyagh

Loading

k8s-ci-robot commented Feb 13, 2021

k8s-ci-robot commented Feb 13, 2021

minikube-bot commented Feb 13, 2021

medyagh commented Feb 17, 2021

ilya-zuyev commented Feb 17, 2021

minikube-pr-bot commented Feb 17, 2021

prezha commented Feb 18, 2021

phantooom commented Feb 18, 2021 •

edited

Loading

check requirements:

examples:

minikube-pr-bot commented Feb 18, 2021

medyagh commented Feb 20, 2021

phantooom commented Feb 20, 2021

priyawadhwa left a comment

prezha commented Feb 26, 2021

medyagh commented Mar 1, 2021

minikube-pr-bot commented Mar 2, 2021

k8s-ci-robot commented Mar 3, 2021

kvm2 driver: Add flag --kvm-numa-count" support topology-manager simulate numa #10471

kvm2 driver: Add flag --kvm-numa-count" support topology-manager simulate numa #10471

Conversation

phantooom commented Feb 13, 2021 • edited by medyagh Loading

k8s-ci-robot commented Feb 13, 2021

k8s-ci-robot commented Feb 13, 2021

minikube-bot commented Feb 13, 2021

medyagh commented Feb 17, 2021

ilya-zuyev commented Feb 17, 2021

minikube-pr-bot commented Feb 17, 2021

prezha commented Feb 18, 2021

check requirements:

examples:

phantooom commented Feb 18, 2021 • edited Loading

check requirements:

examples:

minikube-pr-bot commented Feb 18, 2021

medyagh commented Feb 20, 2021

phantooom commented Feb 20, 2021

priyawadhwa left a comment

Choose a reason for hiding this comment

prezha commented Feb 26, 2021

medyagh commented Mar 1, 2021

minikube-pr-bot commented Mar 2, 2021

k8s-ci-robot commented Mar 3, 2021

phantooom commented Feb 13, 2021 •

edited by medyagh

Loading

phantooom commented Feb 18, 2021 •

edited

Loading