magic-trace hiding perf segmentation fault

[djvolz]

Hey magic-trace Team!

Thank you for all the great effort that's gone into making and supporting magic-trace!

The Issue

On some performance testing machines we've noticed that our perf reports recorded by magic-trace come out fine, but the trace.fxt files are coming out very small (usually 88 bytes).

[root@machine]# magic-trace attach -pid $(pgrep demo)
[ Attached. Press Ctrl-C to stop recording. ]
^C[ Got signal, detaching... ]
[ Snapshot taken. ]
[ perf record: Woken up 737 times to write data ]
Warning:
1 out of order events recorded.
[ perf record: Captured and wrote 587.370 MB /tmp/magic_trace.tmp.f27e6e/perf.data ]
[ Finished recording. ]
[ Decoding, this takes a while... ]

[root@machine]# ls -ls  trace.fxt
4 -rw-r--r-- 1 root root 88 Dec 14 15:15 trace.fxt

The Question

Have you encountered this issue before? Is the decoding step silently failing? If so, do you have any suggestions? Thanks!

Background

What I've checked so far

1. We have debuginfo set up. Symbols show up fine in other programs (gdb, pstack, perf, etc).
2. Happens as sudo or regular user.
3. Perf reports show up correctly on both the older and newer versions of perf.

# Samples: 274  of event 'instructions:u'
# Event count (approx.): 17637701
#
# Children      Self  Command  Shared Object  Symbol                         
# ........  ........  .......  .............  ...............................
#
   100.00%     0.00%  demo     demo           [.] main
            |
            ---main
               |          
               |--76.53%--dlopen@@GLIBC_2.2.5
               |          _dlerror_run
               |          |          
               |           --74.69%--_dl_catch_error
               |                     |          
               |                      --74.69%--dlopen_doit
               |                                |          
               |                                 --74.69%--_dl_open
               |                                           |          
               |                                            --72.78%--_dl_catch_error
               |                                                      |          
               |                                                       --72.78%--dl_open_worker

4. The perf arguments being used by magic-trace are the same (with some variance for the "-t" option) for both the working and non-working machines:

perf record -o /tmp/magic-trace/perf.data --event=intel_pt/cyc=1,cyc_thresh=1,mtc_period=0/u --timestamp --per-thread -t 369397 --snapshot
perf script -i /tmp/magic-trace/perf.data --ns --itrace=be -F pid,tid,time,flags,ip,addr,sym,symoff

5. Turning on MAGIC_TRACE_DEBUG doesn't provide any extra output.
6. The file size does grow larger (to about ~150 bytes) if I turn on -trace-include-kernel or other similar modifications.

Screenshots

Working

Blank

Perf Differences:

I'm wondering if this has something to do with the different versions of perf available on these machines.

• perf version 4.18.0-372.19.1.0.1.el8_6.x86_64
  • Everything works as expected with normal trace.fxt file sizes.
• perf version 4.14.35-2047.511.5.5.1.el7uek.x86_64
  • trace.fxt file always ends up being 88 bytes.

[djvolz]

Update: Looking into this more I found that running the perf script command on the machine with the older version of perf leads to a segmentation fault.

perf script -i /tmp/magic-trace/perf.data --ns --itrace=be -F pid,tid,time,flags,ip,addr,sym,symoff
Segmentation fault

Is there a way magic-trace could propagate this error up to the user?