Unable to run prof when perf_event_paranoid = 2

[murugaratham]

Based on prerequisite

The target machine must have perf_event_paranoid set to 2 or lower.

But i'm unable to run, it's a secured container on eks, which i have limited permissions to change perf_event_paranoid to -1

needs to be less than or equal to 2

cat /proc/sys/kernel/perf_event_paranoid

Dec 20 03:45:39.342061 ddprof[1583]: Error calling perf_event_open on watcher 0.0 (Operation not permitted)
Dec 20 03:45:39.342091 ddprof[1583]: PERFOPEN: error during perf_event_open at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:103
Dec 20 03:45:39.342108 ddprof[1583]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:118 - PERFOPEN: error during perf_event_open
Dec 20 03:45:39.342141 ddprof[1583]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:195 - PERFOPEN: error during perf_event_open
Dec 20 03:45:39.342158 ddprof[1583]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/pevent_lib.cc:264 - PERFOPEN: error during perf_event_open
Dec 20 03:45:39.342175 ddprof[1583]: Forward error at /go/src/github.com/DataDog/apm-reliability/ddprof-build/ddprof/src/ddprof.cc:114 - PERFOPEN: error during perf_event_open
Dec 20 03:45:39.342188 ddprof[1583]: Failed to initialize profiling

xx@container[myapp]:/app# cat /proc/sys/kernel/perf_event_paranoid
2

[murugaratham]

ldd --version
ldd (GNU libc) 2.28
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

uname -r
5.10.228-219.884.amzn2.x86_64

[r1viollet]

Apologies for the late reply.

I suspect that you have extra policies that prevent you from using perf_event_paranoid.
On a simple linux install, perf_event_paranoid=2 is enough to use ddprof in wrapper mode.

Would it be possible to relax the policies to check if this fixes the issue ?

Also, as it could be challenging to adjust these policies, here is an alternative:
It is a host profiler that runs in a separate privileged POD. The nice thing is that it is able to profile several languages (even interpreted languages).

Do tell us if you run into issues setting this up.

[murugaratham]

I looked at the alternatives but sadly it’s a locked down environment and I do not have access to run privileged pods or tools in the nodes(host)

But I did manage to find that the issue is due to MAX_MALLOC_ARENA being left as default and causing fragmentation, I’ve resolved the memory leaks in later deployments.