asan tests fail after build on stretch

[ingwinlu]

Steps to Reproduce the Problem

run elektra-gcc-asan on stretch

Expected Result

It should pass tests the same as on jessie.

Actual Result

Errors are thrown.

System Information

• Elektra Version: master

Further Logs

ASAN Log from a local docker run

[markus2330]

Thank you for reporting! @sanssecours do you understand some of these errors?

[sanssecours]

@sanssecours do you understand some of these errors?

As far as I can tell the errors in testmod_xerces and testmod_botan are caused by problematic code in the Xerces and Botan libraries. To “fix” that problem we could just automatically disable the crypto_botan and xerces plugins on an ASAN enabled GCC build. I have already done that for YAML CPP.

I guess the other failing tests are just a consequence of the two plugins mentioned above throwing errors.

[ingwinlu]

That would also mask problems in the elektra plugins.
Maybe we could introduce a asan suppression file.

[sanssecours]

That would also mask problems in the elektra plugins.

We already have a ASAN enabled Clang build to check for problems.

Maybe we could introduce a asan suppression file.

I think the proper way to suppress errors caused by external libraries is a blacklist file. We already use such a blacklist for Clang. However, as far as I know GCC does not support blacklist files.

[ingwinlu]

I renabled and modified elektra-clang-asan to run on stretch as well.

[ingwinlu]

disabling plugins

Compromising the trustworthiness of a test run just to make it not fail is the Volkswagen approach. Then we don't need suppression/blacklist files and I just edit the job to run sh 'true'. I would prefer to not run a job instead of having one that gives a false positive result.

blacklisting

I think it makes sense to suppress/blacklist errors coming from our dependencies so that they don't fail the tests anymore but still show up when tests are run.

If we can't have sensible suppression/blacklisting of errors with gcc we should consider disabling or removing that build job all together.

[sanssecours]

I would prefer to not run a job instead of having one that gives a false positive result.

There is no false positive test result if we disable a plugin. For example, running testmod_yamlcpp will not work if we use an ASAN enabled GCC build, since the build system will not compile the YAML CPP plugin nor the included tests. The test testmod_yamlcpp will simply not exist.

If we can't have sensible suppression/blacklisting of errors with gcc we should consider disabling or removing that build job all together.

That would work for me. We still have the build job elektra-homepage though, which uses GCC and ASAN.

[ingwinlu]

There is no false positive test result if we disable a plugin.

Maybe I used the wrong words to explain my take on the problem, plus it probably is a bit philosophical.

For me a test run that hides failing tests because they are disabled is not successful.
At the very least the user (in that case the programmer who made the pr) should be very well made aware of that the tests have only run in some configurations (or not run at all).

It probably would be a discussion for another issue to change the existing build setup so that tests are skipped instead of not build to track skipped tests due to misconfiguration in the build system...

[markus2330]

Unfortunately, it is impossible to run all tests in every combination we have. If we disable all plugins which cause a problem on any system, we would have no plugins. And if we disable all agents where any plugin causes a problem, we would have no agent.

So if we want it or not, we will have some situations where some tests (or plugins?) are disabled. I moved the discussion to #1899.

[ingwinlu]

I opened up a PR illustrating the problems with asan in the current build server configuration
#1963

[ingwinlu]

We meanwhile added an asan test and disabled all currently not compatible parts of Elektra to make tests pass.