How to print the function name instead of the address for ustack

[hsueh-yh]

I tried:

bpftrace -e "profile:hz:99 {@stacks[comm,pid,ustack] = count();}"

and got:

...
@stacks[a.out, 172272,
    0x401210
    0x4014c1
    0x4015af
    0x401641
    0x40172b
    0x7f3393041493
    0x5541f689495641d7
]: 2
@stacks[a.out, 172272,
    0x4014d4
    0x4015af
    0x401641
    0x40172b
    0x7f3393041493
    0x5541f689495641d7
]: 2
@stacks[a.out, 172272,
    0x4014bc
    0x4015af
    0x401641
    0x401661
    0x401cb8
    0x401b19
    0x401eec
    0x401ec2
    0x401ea6
    0x7f3393a3fba3
]: 3
......

a.out is a simple c++ demo program compiled with

g++ -g -O0 test.cpp

bpftrace version: v0.13.0
os: centos8 linux 4.18

Expect:
ustack print the function name instead of address.

Question:
Did I missed any compilation parameters or packeges?

Thansk!

[ajor]

Symbolication is based off the symbol table of the target application. It doesn't look like you're doing anything wrong to me, but you could double check that symbols exist in your a.out in a couple of ways:

1. Look for "stripped" or "not stripped" in the output of file a.out.
2. Look at the symbols yourself with nm -p a.out. You'll either see a list of addresses and symbols or the message "no symbols".

If the symbols are fine then another thing to check is whether your process is still running when bpftrace is printing out the stack traces. bpftrace uses the PID to find the correct process to symbolicate against, so if the process no longer exists we won't find the symbol table.

[hsueh-yh]

Thanks for your swift reply! I have checked steps 1 and 2 and can confirm:

1. The symbols have been properly compiled into a.out.
2. The process is a dead loop for testing, so it will always exist.

Another key piece of infomation is that the program runs in a container, and the PID printed by bpftrace is diffrent from the PID printed by ps in the container. Will this cause bpftrace to fail to find the correct process or symbols by PID?

Thanks!

[ajor]

Another key piece of infomation is that the program runs in a container, and the PID printed by bpftrace is diffrent from the PID printed by ps in the container. Will this cause bpftrace to fail to find the correct process or symbols by PID?

Is bpftrace running in the container too? That'll be the problem if so, as the bpftrace userland will see the namespaced PID while the BPF kernel program will get the PID as seen from the init namespace.

I'm actually working on a patch this week which will fix PID collection and symbolication for this case - will update you here when I submit it

[hsueh-yh]

@ajor Has this issue been fixed?

[ajor]

@hsueh-yh I'm so sorry! I completely forgot about this. I'm literally working on the patch right now so hopefully I'll manage to get it finished this time before I get distracted again

[ajor]

I'll have to test it some more, but I've put up a draft fix now: #3428

[baiwfg2]

I have some findings. When I put ustack into map object just like you do, the symbol won't be printed. However, when I just use printf("%s", ustack); ，it will print symbol. NO IDEA why it's that.

[jordalgo]

@baiwfg2 Do you mind pasting your sample script and also the output of bpftrace --info ?

Ah, nm, I see your new discussion topic now: #3614