Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add eBPF related kernel options #14316

Merged
merged 4 commits into from
Jun 14, 2022

Conversation

eiffel-fl
Copy link
Contributor

Hi.

This PR permits using minikube 5.10 kernel to develop eBPF related feature.
It follows #12707.

I use it as my daily driver and I know developers from Polar Signals use it too.

I was able to build the images and run them:

# For x86_64
francis@pwmachine:~/Codes/kinvolk$ minikube start --driver=kvm --iso-url=file:///home/francis/Codes/kinvolk/minikube-amd64.iso    
😄  minikube v1.23.2 sur Ubuntu 20.04
✨  Utilisation du pilote kvm2 basé sur la configuration de l'utilisateur
👍  Démarrage du noeud de plan de contrôle minikube dans le cluster minikube
🔥  Création de VM kvm2 (CPUs=2, Mémoire=6000MB, Disque=20000MB)...
🐳  Préparation de Kubernetes v1.22.2 sur Docker 20.10.16...
    ▪ Génération des certificats et des clés
    ▪ Démarrage du plan de contrôle ...
    ▪ Configuration des règles RBAC ...
🔎  Vérification des composants Kubernetes...
    ▪ Utilisation de l'image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Modules activés: storage-provisioner, default-storageclass
🏄  Terminé ! kubectl est maintenant configuré pour utiliser "minikube" cluster et espace de noms "default" par défaut.
francis@pwmachine:~/Codes/kinvolk$ minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ uname -a
Linux minikube 5.10.57 #1 SMP Fri Jun 10 14:03:59 UTC 2022 x86_64 GNU/Linux
$ zcat /proc/config.gz | grep BTF
CONFIG_DEBUG_INFO_BTF=y
$ ls /sys/kernel/btf/
vmlinux
# For arm64, I tested without minikube because I do not have an arm64 hardware
# For arm64:
francis@pwmachine:~/Codes/kinvolk$ wget https://releases.linaro.org/components/kernel/uefi-linaro/16.02/release/qemu64/QEMU_EFI.img.gz
# ...
francis@pwmachine:~/Codes/kinvolk$ gunzip QEMU_EFI.img.gz
# QEMU_EFI contains grub or arm64.
francis@pwmachine:~/Codes/kinvolk$ qemu-system-aarch64 -cpu cortex-a53 -M virt -m 4096 -nographic -pflash QEMU_EFI.img -hda minikube-arm64.iso
# Some systemd failures are printed:
[FAILED] Failed to listen on Docker Socket for the API.
See 'systemctl status docker.socket' for details.
[FAILED] Failed to start minikube scheduled stop.
See 'systemctl status minikube-scheduled-stop.service' for details.
[FAILED] Failed to start CRI-O Auto Update Script.
See 'systemctl status crio-wipe.service' for details.
[DEPEND] Dependency failed for Cont…ime Interface for OCI (CRI-O).
[  OK  ] Started User Login Management.

Welcome to minikube
minikube login: root
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

# uname -a
Linux minikube 5.10.57 #1 SMP PREEMPT Fri Jun 10 14:54:37 UTC 2022 aarch64 GNU/Linux
# zcat /proc/config.gz | grep BTF
CONFIG_DEBUG_INFO_BTF=y

The implementation of this feature is not perfect (particularly the backporting of upstream buildroot patch) but I think this is the only solution we have since some upstream patches are not part of last buildroot LTS.

Best regards and thank you in advance.

iso-menuconfig would do nothing related to buildroot configurations with the
changes added with arm64.

Signed-off-by: Francis Laniel <flaniel@linux.microsoft.com>
This package is needed to build the kernel when using CONFIG_DEBUG_INFO_BTF.

Signed-off-by: Francis Laniel <flaniel@linux.microsoft.com>
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jun 10, 2022

CLA Signed

The committers listed above are authorized under a signed CLA.

@k8s-ci-robot k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jun 10, 2022
@k8s-ci-robot
Copy link
Contributor

Hi @eiffel-fl. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 10, 2022
@minikube-bot
Copy link
Collaborator

Can one of the admins verify this patch?

This config option permits indicating the Linux kernel needs pahole to be
compiled.
Thus, host-pahole will be built before the kernel.
This is mandatory when using CONFIG_DEBUG_INFO_BTF.

Signed-off-by: Francis Laniel <flaniel@microsoft.com>
Signed-off-by: Francis Laniel <flaniel@linux.microsoft.com>
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 10, 2022
@medyagh
Copy link
Member

medyagh commented Jun 13, 2022

ok-to-build-iso

@minikube-bot
Copy link
Collaborator

Hi @eiffel-fl, we have updated your PR with the reference to newly built ISO. Pull the changes locally if you want to test with them or update your PR further.

@spowelljr
Copy link
Member

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 14, 2022
@minikube-pr-bot
Copy link

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14316) |
+----------------+----------+---------------------+
| minikube start | 51.7s    | 52.3s               |
| enable ingress | 27.4s    | 27.0s               |
+----------------+----------+---------------------+

Times for minikube start: 51.3s 51.2s 52.1s 52.3s 51.6s
Times for minikube (PR 14316) start: 52.4s 52.0s 50.7s 52.7s 54.0s

Times for minikube ingress: 28.1s 29.6s 28.6s 25.1s 25.6s
Times for minikube (PR 14316) ingress: 29.1s 25.0s 28.6s 26.1s 26.1s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14316) |
+----------------+----------+---------------------+
| minikube start | 29.6s    | 24.1s               |
| enable ingress | 22.8s    | 22.6s               |
+----------------+----------+---------------------+

Times for minikube (PR 14316) start: 23.7s 24.5s 24.2s 23.7s 24.4s
Times for minikube start: 50.5s 24.0s 24.1s 25.0s 24.3s

Times for minikube ingress: 21.9s 25.9s 21.9s 22.5s 22.0s
Times for minikube (PR 14316) ingress: 22.4s 22.9s 21.9s 23.9s 22.0s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 14316) |
+----------------+----------+---------------------+
| minikube start | 31.6s    | 30.4s               |
| enable ingress | 21.6s    | 25.0s               |
+----------------+----------+---------------------+

Times for minikube start: 29.8s 33.1s 31.9s 32.1s 31.0s
Times for minikube (PR 14316) start: 29.1s 32.7s 32.1s 29.1s 29.1s

Times for minikube ingress: 32.4s 18.9s 18.9s 18.9s 18.9s
Times for minikube (PR 14316) ingress: 22.4s 28.9s 18.9s 22.4s 32.5s

@minikube-pr-bot
Copy link

These are the flake rates of all failed tests.

Environment Failed Tests Flake Rate (%)
Docker_macOS TestDownloadOnlyKic (gopogh) 7.80 (chart)
Docker_macOS TestDownloadOnly/v1.23.6/preload-exists (gopogh) 7.80 (chart)
Docker_macOS TestFunctional/parallel/DashboardCmd (gopogh) 20.63 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/DeployApp (gopogh) 21.88 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/UserAppExistsAfterStop (gopogh) 21.88 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/FirstStart (gopogh) 22.68 (chart)
Docker_Linux_containerd TestStartStop/group/no-preload/serial/SecondStart (gopogh) 22.68 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh) 24.21 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/SecondStart (gopogh) 24.21 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh) 25.00 (chart)
Docker_Linux_containerd TestStartStop/group/old-k8s-version/serial/UserAppExistsAfterStop (gopogh) 25.00 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/DeployApp (gopogh) 27.37 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/FirstStart (gopogh) 28.12 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/SecondStart (gopogh) 28.12 (chart)
Docker_Linux_containerd TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh) 28.12 (chart)
Hyperkit_macOS TestSkaffold (gopogh) 36.31 (chart)
Docker_Linux TestFunctional/serial/ComponentHealth (gopogh) 36.97 (chart)
Docker_macOS TestSkaffold (gopogh) 38.89 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/DeployApp (gopogh) 45.26 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/UserAppExistsAfterStop (gopogh) 45.26 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/SecondStart (gopogh) 45.83 (chart)
Docker_Linux_containerd TestStartStop/group/default-k8s-different-port/serial/FirstStart (gopogh) 46.94 (chart)
Docker_Windows TestMinikubeProfile (gopogh) 48.03 (chart)
Docker_Windows TestAddons/Setup (gopogh) 48.67 (chart)
Docker_Windows TestFunctional/parallel/CertSync (gopogh) 48.67 (chart)
Docker_Windows TestFunctional/parallel/CpCmd (gopogh) 48.67 (chart)
Docker_Windows TestFunctional/parallel/DockerEnv/powershell (gopogh) 48.67 (chart)
Docker_Windows TestFunctional/parallel/FileSync (gopogh) 48.67 (chart)
Docker_Windows TestFunctional/parallel/ImageCommands/ImageBuild (gopogh) 48.67 (chart)
Docker_Windows TestFunctional/parallel/ImageCommands/ImageListJson (gopogh) 48.67 (chart)
More tests... Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

@spowelljr spowelljr merged commit 7e65323 into kubernetes:master Jun 14, 2022
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: eiffel-fl, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 14, 2022
@eiffel-fl eiffel-fl deleted the alban/kernel-5.10-options branch June 15, 2022 09:41
@eiffel-fl
Copy link
Contributor Author

Thank you a lot for the merge :D :D :D!

@kakkoyun
Copy link
Contributor

I'm ecstatic that this has been merged! Thanks a lot, @eiffel-fl, for the amazing work!

Can we just use it? Will there be a release candidate that includes new Kernel upgrades? Or is there an easy way to grab an image from the merged commits?

@victorsndvg
Copy link

Hi @ALL,

I'm trying to run ebpf programs with the latest minikube version (1.26.0), but it is not working.

$ minikube ssh -- docker run --rm --privileged -v /lib/modules:/lib/modules:ro -v /usr/src:/usr/src:ro -v /etc/localtime:/etc/localtime:ro --workdir /usr/share/bcc/tools zlim/bcc ./execsnoop
chdir(/lib/modules/5.10.57/build): No such file or directory
Traceback (most recent call last):
  File "/usr/share/bcc/tools/execsnoop", line 165, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 325, in __init__
    raise Exception("Failed to compile BPF text")
Exception: Failed to compile BPF text
ssh: Process exited with status 1
$ minikube ssh -- uname -a
Linux minikube 5.10.57 #1 SMP Thu Jun 16 23:36:20 UTC 2022 x86_64 GNU/Linux
$ minikube ssh -- ls /lib/modules/5.10.57/
Module.symvers	modules.alias	   modules.builtin.alias.bin  modules.dep      modules.order	modules.symbols.bin
extra		modules.alias.bin  modules.builtin.bin	      modules.dep.bin  modules.softdep
kernel		modules.builtin    modules.builtin.modinfo    modules.devname  modules.symbols

I suppose it should work. Maybe, I'm missunderstunding something?

I really appreciate your help.

Thanks in advance.
Víctor.

@eiffel-fl
Copy link
Contributor Author

eiffel-fl commented Jul 11, 2022

Hi.

I tested your case and was indeed able to reproduce.
Nonetheless, the problem is more tied to the docker image you use which has not been updated since 3 years!
With the official image (which was updated 5 months ago), I am able to run execsnoop:

$ minikube start --driver=kvm
...
$ minikube ssh
$ docker run --rm --privileged -v /lib/modules:/lib/modules:ro -v /usr/src:/usr/src:ro -v /etc/localtime:/etc/localtime:ro --workdir /usr/share/bcc/tools quay.io/iovisor/bcc ./execsnoop     
Unable to find image 'quay.io/iovisor/bcc:latest' locally
latest: Pulling from iovisor/bcc
68e7bb398b9f: Pull complete 
aa73686fdcd4: Pull complete 
7e88b7dfb0a8: Pull complete 
Digest: sha256:63f8262abfa9e8fc531f23c960b27736c75d1f13fff20d9c34d3387391232dd9
Status: Downloaded newer image for quay.io/iovisor/bcc:latest
PCOMM            PID    PPID   RET ARGS
runc             5067   920      0 /usr/bin/runc --version
docker-init      5072   920      0 /usr/bin/docker-init --version
iptables         5073   2273     0 /usr/sbin/iptables -w 5 -W 100000 -S KUBE-PROXY-CANARY -t mangle
runc             5074   920      0 /usr/bin/runc --version
docker-init      5079   920      0 /usr/bin/docker-init --version
runc             5080   920      0 /usr/bin/runc --version
runc             5081   920      0 /usr/bin/runc --version
runc             5087   920      0 /usr/bin/runc --version
runc             5088   920      0 /usr/bin/runc --version
runc             5092   920      0 /usr/bin/runc --version
runc             5101   920      0 /usr/bin/runc --version
runc             5107   920      0 /usr/bin/runc --version
...

Also, please be aware than "bcc based" eBPF tools (also known as "standard") are a bit old and are not the more recent way to interact with eBPF.
Indeed, the more recent CO-RE eBPF tools are easier and more flexible to use; this PR was mainly about adding kernel options needed by CO-RE tools.
Think of standard tools as python 2 and CO-RE as python3.

I hope it solves your problem, otherwise feel free to ping.

Best regards.

@victorsndvg
Copy link

Thanks @eiffel-fl,

you are right, the proposed docker image is working as expected.

Sorry for bothering you, but there are a lot of old info about eBPF and I am lost ...

I will take a look to CO-RE tools.

Thanks again! :)
Víctor

@eiffel-fl
Copy link
Contributor Author

Thanks @eiffel-fl,

You are welcome :).

Sorry for bothering you, but there are a lot of old info about eBPF and I am lost ...

No problem, it is a bit normal to be lost when learning eBPF.
The following link helped me to understand the difference (and transition) from standard to CO-RE tools:
https://nakryiko.com/posts/bcc-to-libbpf-howto-guide/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants