Aggregate perf metrics

[katarzyna-z]

Signed-off-by: Katarzyna Kujawa katarzyna.kujawa@intel.com

I want to test it more and add more unit tests.
Related to #2608

[katarzyna-z]

Tested with 8 cores, 60 containers, 5 perf events. configuration file:

{
  "core": {
    "events": [
      ["instructions"],
      ["instructions_retired"],
      ["cycles"],
      ["ref-cycles"],
      ["context-switches"]
    ],
    "custom_events": [
      {
        "type": 4,
        "config": [
          "0x5300c0"
        ],
        "name": "instructions_retired"
      }
    ]
  }
}

Aggregated perf events
command: sudo ./cadvisor -perf_events_config=perf/testing/perf.json -disable_metrics="accelerator,disk,diskIO,network,process,hugetlb,resctrl,udp,advtcp,referenced_memory,cpu_topology,tcp,sched,percpu"
response time on /metrics endpoint: 0.419s
data size of response: 5MB

Per CPU perf events
command: sudo ./cadvisor -perf_events_config=perf/testing/perf.json -disable_metrics="accelerator,disk,diskIO,network,process,hugetlb,resctrl,udp,advtcp,referenced_memory,cpu_topology,tcp,sched"
response time on /metrics endpoint: 0.514s
data size of response: 14MB

@wacuuu Could you test this pull request in your environment?

[wacuuu]

baseline for disable_metrics is set to
udp,referenced_memory,resctrl,hugetlb,cpu_topology,tcp,advtcp,sched,process, i only add or remove percpu parameter

To measure the parameters i run the following command on the node that is running cadvisor in container:
time curl localhost:8080/metrics > /dev/null

Number of containers	Number of enabled perf counters	Time of scraping(s)	Scraped data volume(MB)	disable percpu
58	5	0.89s	20M	yes
58	5	3.07s	97M	no
--	--	--	--	--
58	10	1.82s	21.8M	yes
58	10	12.92s	168M	no
--	--	--	--	--
32	10	0.86s	15.3M	yes
32	10	4.81s	21.4M	no

So the conclusion would be that percpu significantly improves the situation.

[katarzyna-z]

@dashpole it is ready for review.

[ppalucki]

IMO "average" is no useful comparing to the worst case when scaling happened
after aggregation with average equal 90% one doesnt know if the all cpus were scaled with by 10% or most of them were 100% and one was running only 5% and scaled 20 times (quite significant error)

You should add or replace (avg) with min scaling ratio accross all cpus.

[katarzyna-z]

ok so I'll replace average with min

[katarzyna-z]

/retest

[dashpole]

@katarzyna-z is it possible to unit test this?

[katarzyna-z]

@dashpole Unit tests are available in this file. Do you think that I should add something?

[dashpole]

lgtm