CE log is flooded with DEBUG and ERROR entries

[johaag]

Description

After updating to sonar-cxx v1.2.0 on SonarQube 6.7.6 LTS Enterprise Edition, a lot of DEBUG and ERROR messages occur in the CE log.

During portfolio calculation (VIEW_REFRESH) for each portfolio calculated the following log entries are created (log level DEBUG):

2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.s.c.t.s.ComputationStepExecutor] Load measure computers | time=0ms
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-PUBLIC_API is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-PUBLIC_UNDOCUMENTED_API is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-CLANGSA is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-CLANG-TIDY is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-COMPILER-VC is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-COMPILER-GCC is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-CPPCHECK is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-DRMEMORY is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-OTHER is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-PC-LINT is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-RATS is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-SQUID is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-VALGRIND is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-VERA++ is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-COMPLEX_FUNCTIONS is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-LOC_IN_COMPLEX_FUNCTIONS is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-BIG_FUNCTIONS is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-LOC_IN_BIG_FUNCTIONS is not set for children
2018.12.21 14:31:34 DEBUG ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.AggregateMeasureComputer] Component Portfolio_XXX: measure CXX-LOC_IN_FUNCTIONS is not set for children
2018.12.21 14:31:34 ERROR ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.DensityMeasureComputer] Component Portfolio_XXX: not enough data to calcualte measure CXX-PUBLIC_DOCUMENTED_API_DENSITY
2018.12.21 14:31:34 ERROR ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.DensityMeasureComputer] Component Portfolio_XXX: not enough data to calcualte measure CXX-PERC_COMPLEX_FUNCTIONS
2018.12.21 14:31:34 ERROR ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.DensityMeasureComputer] Component Portfolio_XXX: not enough data to calcualte measure CXX-PERC_LOC_IN_COMPLEX_FUNCTIONS
2018.12.21 14:31:34 ERROR ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.DensityMeasureComputer] Component Portfolio_XXX: not enough data to calcualte measure CXX-PERC_BIG_FUNCTIONS
2018.12.21 14:31:34 ERROR ce[AWfQ9WJCyheR3_yW1sX3][o.s.c.DensityMeasureComputer] Component Portfolio_XXX: not enough data to calcualte measure CXX-PERC_LOC_IN_BIG_FUNCTIONS

These log entries are created also for empty portfolios and for portfolios containing projects which do not use sonar-cxx.

Similar entries are created during processing of scan results (REPORT):

2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.s.c.t.s.ComputationStepExecutor] Load measure computers | time=0ms
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 DEBUG ce[AWe70cHKyheR3_yW1n7H][o.s.c.AggregateMeasureComputer] Component project_key:script/file.py: FILE doesn't required an aggregation
2018.12.17 11:59:10 ERROR ce[AWe70cHKyheR3_yW1n7H][o.s.c.DensityMeasureComputer] Component project_key:script/file.py: not enough data to calcualte measure CXX-PUBLIC_DOCUMENTED_API_DENSITY
2018.12.17 11:59:10 ERROR ce[AWe70cHKyheR3_yW1n7H][o.s.c.DensityMeasureComputer] Component project_key:script/file.py: not enough data to calcualte measure CXX-PERC_COMPLEX_FUNCTIONS
2018.12.17 11:59:10 ERROR ce[AWe70cHKyheR3_yW1n7H][o.s.c.DensityMeasureComputer] Component project_key:script/file.py: not enough data to calcualte measure CXX-PERC_LOC_IN_COMPLEX_FUNCTIONS
2018.12.17 11:59:10 ERROR ce[AWe70cHKyheR3_yW1n7H][o.s.c.DensityMeasureComputer] Component project_key:script/file.py: not enough data to calcualte measure CXX-PERC_BIG_FUNCTIONS
2018.12.17 11:59:10 ERROR ce[AWe70cHKyheR3_yW1n7H][o.s.c.DensityMeasureComputer] Component project_key:script/file.py: not enough data to calcualte measure CXX-PERC_LOC_IN_BIG_FUNCTIONS

These log entries are created for each file of a project, regardless of the language and if sonar-cxx is used by the project.

This is a lot of noise in the CE log file and especially the ERRORs are a problem from our point of view.

Steps to reproduce the problem

Please provide the steps required to reproduce the problem

1. Install SonarQube 6.7.6 LTS Enterprise Edition along with sonar-cxx 1.2.0

2. Set log level for CE to DEBUG

3. Trigger portfolio calculation via Web API api/views/run

4. Scan a project and upload it to SonarQube

5. Observe CE log entries in logs/ce.log

Expected behavior

Log entries should only be created for files and projects which are analysed by sonar-cxx.

Actual behavior

Log entries are created for any file and project.

Known workarounds

None.

LOG file

See above.

Related information

• cxx plugin version 1.2.0
• SonarQube version 6.7.6 LTS Enterprise Edition

[ivangalkin]

@johaag thank you for your report.

1. your sonar.log.level.ce logging level is lowered. It looks like you enabled diagnostic logging (DEBUG), so no wonder there are so many message... but
2. you are absolutely correct about the ERROR entries: our MeasureComputer might visit components, which were not analyzed by sonar-cxx. We should not complain about some missing data in such cases.

@guwirth
There is no easy way to determine if e.g. the Module or Directory were analyzed by our plugin. We cannot be sure about the error situation in such case.

I suggest just to reduce the logging amount. E.g. UnitTestMeasuresStep doesn't log at all (although it's based on some other [internal] technology).

I'll provide the PR. I believe it would match the 1.2.1 milestone well. "optimize loggin" is one of its main aims.

[johaag]

@ivangalkin thank you so much for taking care of this issue and fixing it so promptly! Your change looks good to me. Great to see that we will get a fix in 1.2.1.

[guwirth]

@johaag you can try with latest snapshot.

[johaag]

@ivangalkin and @guwirth, checked with latest snapshot. LGTM. Thanks a lot for taking care!