engine.report_pmu() [WIP]

[lukego]

This PR is an alternative to #615 for tracking performance counter values per app.

This branch counts the performance events during each individual push() and pull() call and accumulates a total for each app. That is to say that it directly counts the number cycles, cache misses, etc, that occur during each callback.

The function engine.report_pmu() prints the values for each app and also calculates per-packet values. (The number of packets the apps has processed is determined by its links.)

The event counting seems to work surprisingly well. I was concerned that the individual callbacks would be too short and so there would be too much noise when trying to count their PMU events. However, the initial results are very consistent with the numbers reported in #615 based on long running averages.

Results for a Tee app in a Source->Tee->Sink network from this branch:

$ sudo taskset -c 0 ./snabb snabbmark basic1 1e9
...
*** Tee
EVENT                                             TOTAL     /packet
cycles                                   14,784,901,928      14.780
ref_cycles                               11,104,338,216      11.100
instructions                             36,742,279,315      36.729
br_misp_retired.all_branches                  3,992,662       0.004
mem_load_uops_retired.l1_hit             12,783,402,186      12.779
mem_load_uops_retired.l2_hit                991,799,311       0.991
mem_load_uops_retired.l3_hit                     61,993       0.000
mem_load_uops_retired.l3_miss                         0       0.000
packet                                    1,000,359,900       1.000

and from #615 with comparable /packet column:

difference from reference to production:
EVENT                                             TOTAL     /packet
cycles                                    1,409,813,298      14.098
ref_cycles                                1,056,440,016      10.564
instructions                              3,858,706,478      38.587
br_misp_retired.all_branches                    361,693       0.004
mem_load_uops_retired.l1_hit              1,324,505,106      13.245
mem_load_uops_retired.l2_hit                109,539,516       1.095
mem_load_uops_retired.l3_hit                    180,222       0.002
mem_load_uops_retired.l3_miss                         0       0.000
packet                                      100,000,000       1.000

The overhead is significant however and the basic1 benchmark loses around 1/3 of its throughput when sampling the PMU counters. So you would only enable this feature when you are willing to sacrifice overall performance to see per-app performance.

Could be a good plan to tidy up this code and merge it in preference to #615.

I am still interested in having an appbench style program that can generate a "datasheet" for an app that estimates how it will perform in different situations (packet size, traffic mix, data in L1/L2/L3/DRAM, etc). However, it should be possible to build that on this code anyway.

[lukego]

Closing PR: I am not actively developing this branch.