Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

invalid memory address or nil pointer dereference #1428

Closed
rgarcia89 opened this issue Sep 5, 2023 · 5 comments · Fixed by #1432
Closed

invalid memory address or nil pointer dereference #1428

rgarcia89 opened this issue Sep 5, 2023 · 5 comments · Fixed by #1432
Assignees
@tpapagian
Labels
kind/bug Something isn't working

Comments

@rgarcia89
Copy link

What happened?

  1. AKS
  2. helm chart with the following values
USER-SUPPLIED VALUES:
nodeSelector:
  kubernetes.azure.com/mode: user
tetragon:
  exportAllowList: '{"event_set":["PROCESS_EXEC"]}'
  exportDenyList: |-
    {"health_check":true}
    {"namespace":["", "calico-system", "elastic-system", "kube-system", "kyverno", "monitoring", "tigera-operator"]}
  fieldFilters: '{"fields": "parent.cwd,parent.binary,parent.arguments,parent.flags,parent.pod.container,parent.pod.name,parent.pod.namespace",
    "action": "INCLUDE"}'
  grpc:
    enabled: false
  prometheus:
    address: ""
    enabled: false
    port: 2112
    serviceMonitor:
      enabled: true
      labelsOverride: {}

Tetragon Version

v0.11.0

Kernel Version

5.15.116.1-2.cm2

Kubernetes Version

1.25.11

Bugtool

/ # tetra bugtool
time="2023-09-05T09:01:00Z" level=info msg="saving init info"
time="2023-09-05T09:01:00Z" level=info msg="retrieving lib directory" libDir=/var/lib/tetragon/
time="2023-09-05T09:01:00Z" level=warning msg="not an object file, ignoring" path=/var/lib/tetragon/
time="2023-09-05T09:01:01Z" level=info msg="skipping metadata directory" path=/var/lib/tetragon/metadata
time="2023-09-05T09:01:01Z" level=warning msg="no btf filename in tetragon config, attempting to fall back to /sys/kernel/btf/vmlinux"
time="2023-09-05T09:01:02Z" level=info msg="btf file added" btfFname=/sys/kernel/btf/vmlinux
time="2023-09-05T09:01:02Z" level=info msg="tetragon log file added" exportFname=/var/run/cilium/tetragon/tetragon.log
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd=/bin/dmesg dstFname=dmesg.out ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev lo ingress" dstFname=tc-info.lo.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev lo egress" dstFname=tc-info.lo.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev eth0 ingress" dstFname=tc-info.eth0.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev eth0 egress" dstFname=tc-info.eth0.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali52a51b7e150 ingress" dstFname=tc-info.cali52a51b7e150.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali52a51b7e150 egress" dstFname=tc-info.cali52a51b7e150.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calia6756a6d022 ingress" dstFname=tc-info.calia6756a6d022.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calia6756a6d022 egress" dstFname=tc-info.calia6756a6d022.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calibb1c3c9c54b ingress" dstFname=tc-info.calibb1c3c9c54b.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calibb1c3c9c54b egress" dstFname=tc-info.calibb1c3c9c54b.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali2195b1f9100 ingress" dstFname=tc-info.cali2195b1f9100.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali2195b1f9100 egress" dstFname=tc-info.cali2195b1f9100.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali8e6138bc8ef ingress" dstFname=tc-info.cali8e6138bc8ef.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali8e6138bc8ef egress" dstFname=tc-info.cali8e6138bc8ef.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calie75be914d51 ingress" dstFname=tc-info.calie75be914d51.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calie75be914d51 egress" dstFname=tc-info.calie75be914d51.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calid63d52b1ea2 ingress" dstFname=tc-info.calid63d52b1ea2.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calid63d52b1ea2 egress" dstFname=tc-info.calid63d52b1ea2.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali9822dfd3433 ingress" dstFname=tc-info.cali9822dfd3433.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali9822dfd3433 egress" dstFname=tc-info.cali9822dfd3433.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calibe095208499 ingress" dstFname=tc-info.calibe095208499.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calibe095208499 egress" dstFname=tc-info.calibe095208499.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali71566d78567 ingress" dstFname=tc-info.cali71566d78567.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali71566d78567 egress" dstFname=tc-info.cali71566d78567.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali92bdcbdcbc3 ingress" dstFname=tc-info.cali92bdcbdcbc3.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali92bdcbdcbc3 egress" dstFname=tc-info.cali92bdcbdcbc3.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali758e1cdec7c ingress" dstFname=tc-info.cali758e1cdec7c.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali758e1cdec7c egress" dstFname=tc-info.cali758e1cdec7c.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali081fa9c11be ingress" dstFname=tc-info.cali081fa9c11be.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali081fa9c11be egress" dstFname=tc-info.cali081fa9c11be.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali5352086ef57 ingress" dstFname=tc-info.cali5352086ef57.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev cali5352086ef57 egress" dstFname=tc-info.cali5352086ef57.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev califebd4108b03 ingress" dstFname=tc-info.califebd4108b03.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev califebd4108b03 egress" dstFname=tc-info.califebd4108b03.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev eth1 ingress" dstFname=tc-info.eth1.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev eth1 egress" dstFname=tc-info.eth1.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calie88c039dd3f ingress" dstFname=tc-info.calie88c039dd3f.ingress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/sbin/tc filter show dev calie88c039dd3f egress" dstFname=tc-info.calie88c039dd3f.egress ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/usr/bin/bpftool map show -j" dstFname=bpftool-maps.json ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/usr/bin/bpftool prog show -j" dstFname=bpftool-progs.json ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/usr/bin/bpftool cgroup tree -j" dstFname=bpftool-cgroups.json ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/usr/bin/gops stack localhost:8118" dstFname=gops.stack ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/usr/bin/gops stats localhost:8118" dstFname=gpos.stats ret=0
time="2023-09-05T09:01:02Z" level=info msg="executed command" cmd="/usr/bin/gops memstats localhost:8118" dstFname=gops.memstats ret=0
time="2023-09-05T09:01:02Z" level=warning msg="failed to open policyfilter map" error="no such file or directory"
time="2023-09-05T09:01:02Z" level=info msg="dumped tracing policies in tracing-policies.json"
/ # command terminated with exit code 137

Relevant log output

time="2023-09-05T08:57:30Z" level=info msg="Listening for events..."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2212a7b]

goroutine 101 [running]:
github.com/cilium/tetragon/pkg/grpc/exec.GetProcessExit(0xc001fec630)
        /go/src/github.com/cilium/tetragon/pkg/grpc/exec/exec.go:301 +0x25b
github.com/cilium/tetragon/pkg/grpc/exec.(*MsgExitEventUnix).HandleMessage(0xc001fec630)
        /go/src/github.com/cilium/tetragon/pkg/grpc/exec/exec.go:370 +0x36
github.com/cilium/tetragon/pkg/grpc.(*ProcessManager).Notify(0x40e087?, {0x2d5c640, 0xc001fec630})
        /go/src/github.com/cilium/tetragon/pkg/grpc/process_manager.go:59 +0x2c
github.com/cilium/tetragon/pkg/observer.(*Observer).observerListeners(0xc000a81f80, {0x2d5c640, 0xc001fec630})
        /go/src/github.com/cilium/tetragon/pkg/observer/observer.go:55 +0xb2
github.com/cilium/tetragon/pkg/observer.(*Observer).receiveEvent(0xc000a81f80, {0xc001fec5a0?, 0x14c?, 0x14c?})
        /go/src/github.com/cilium/tetragon/pkg/observer/observer.go:147 +0x3c8
github.com/cilium/tetragon/pkg/observer.(*Observer).RunEvents.func1()
        /go/src/github.com/cilium/tetragon/pkg/observer/observer.go:229 +0x145
created by github.com/cilium/tetragon/pkg/observer.(*Observer).RunEvents
        /go/src/github.com/cilium/tetragon/pkg/observer/observer.go:216 +0x38a

Anything else?

Pods are the in crashloopbackoff

pod/tetragon-42h2j 1/2 CrashLoopBackOff 3 (37s ago) 4m26s 10.34.28.76 aks-workerpool1-13479091-vmss000001
pod/tetragon-gqj6t 1/2 Error 4 (92s ago) 4m26s 10.34.28.72 aks-workerpool1-13479091-vmss000002
pod/tetragon-sb299 2/2 Running 3 (35s ago) 4m26s 10.34.28.70 aks-workerpool1-13479091-vmss000000

@rgarcia89 rgarcia89 added the kind/bug Something isn't working label Sep 5, 2023
@mtardy
Copy link
Member

mtardy commented Sep 5, 2023

Thanks for the report, we will take a look.

@tpapagian
Copy link
Member

Hello, thanks for the report. I managed to reproduce that.

I found a quick workaround. If you add parent.pid in fieldFilters this seems to fix the issue (at least in my setup). i.e.:

fieldFilters: '{"fields": "parent.pid,parent.cwd,parent.binary,parent.arguments,parent.flags,parent.pod.container,parent.pod.name,parent.pod.namespace",
    "action": "INCLUDE"}'

It would be great if you could confirm that.

In the meantime, I will work on a more proper fix (i.e. allowing fieldFilters that do not include parent.pid).

@rgarcia89
Copy link
Author

I can confirm adding parent.pid fixes the issue 👍

tpapagian added a commit that referenced this issue Sep 5, 2023
@tpapagian
Use a message copy to apply fieldFilters in exec events
a81851a 
As we keep them in the process cache, removing fields may affect the
correctness of the process cache. In order to fix that, we create a copy
of the event and apply fieldFilters only on the copy.

FIXES: #1428

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
@tpapagian tpapagian mentioned this issue Sep 5, 2023
Merged
tpapagian added a commit that referenced this issue Sep 5, 2023
@tpapagian
Use a message copy to apply fieldFilters in exec events
0d85e0d 
As we keep them in the process cache, removing fields may affect the
correctness of the process cache. In order to fix that, we create a copy
of the event and apply fieldFilters only on the copy.

FIXES: #1428

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
tpapagian added a commit that referenced this issue Sep 6, 2023
@tpapagian
Use a message copy to apply fieldFilters in exec events
dca73e3 
As we keep them in the process cache, removing fields may affect the
correctness of the process cache. In order to fix that, we create a copy
of the event and apply fieldFilters only on the copy.

FIXES: #1428

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
@tpapagian tpapagian self-assigned this Sep 6, 2023
@tpapagian
Copy link
Member

The fix is merged: #1432

This will be part of the next release.

@tpapagian tpapagian reopened this Sep 6, 2023
@tpapagian
Copy link
Member

Closing as this is fixed in the main branch (and latest images i.e. quay.io/cilium/tetragon-operator-ci and quay.io/cilium/tetragon-ci). Please re-open if this is still an issue.

willfindlay pushed a commit that referenced this issue Oct 31, 2023
@tpapagian @willfindlay
Use a message copy to apply fieldFilters in exec events
8f6a6a5 
As we keep them in the process cache, removing fields may affect the
correctness of the process cache. In order to fix that, we create a copy
of the event and apply fieldFilters only on the copy.

FIXES: #1428

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
willfindlay pushed a commit that referenced this issue Oct 31, 2023
@tpapagian @willfindlay
Use a message copy to apply fieldFilters in exec events
c005a7d 
[upstream commit: dca73e3]

As we keep them in the process cache, removing fields may affect the
correctness of the process cache. In order to fix that, we create a copy
of the event and apply fieldFilters only on the copy.

FIXES: #1428

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
@willfindlay willfindlay mentioned this issue Oct 31, 2023
Merged
willfindlay pushed a commit that referenced this issue Oct 31, 2023
@tpapagian @willfindlay
Use a message copy to apply fieldFilters in exec events
5a43f7e 
[upstream commit: dca73e3]

As we keep them in the process cache, removing fields may affect the
correctness of the process cache. In order to fix that, we create a copy
of the event and apply fieldFilters only on the copy.

FIXES: #1428

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
willfindlay pushed a commit that referenced this issue Oct 31, 2023
@tpapagian @willfindlay
Use a message copy to apply fieldFilters in exec events
697dd77 
[upstream commit: dca73e3]

As we keep them in the process cache, removing fields may affect the
correctness of the process cache. In order to fix that, we create a copy
of the event and apply fieldFilters only on the copy.

FIXES: #1428

Signed-off-by: Anastasios Papagiannis <tasos.papagiannnis@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tpapagian tpapagian

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use a message copy to apply fieldFilters in exec events cilium/tetragon
3 participants
@tpapagian @rgarcia89 @mtardy