Mull is not mutating the shared object file

[duthades]

The tool is not mutating the shared object file. Here are the details to replicate the behavior.

Details:

valid_age.h

#ifndef MULL_EXAMPLE_H
#define MULL_EXAMPLE_H

int valid_age(int age);

#endif

valid_age.cpp

#include "valid_age.h"

int valid_age(int age) {
    if (age >= 21) {
        return 1;
    }
    return 0;
}

test_valid_age.cpp

#include "mull_example_lib.h"


int main() {
    int test1 = valid_age(25) == true;
    if (!test1) {
        /// test failed
        return 1;
    }

    int test2 = valid_age(20) == false;
    if (!test2) {
        /// test failed
        return 1;
    }

    /// success
    return 0;
}

Commands

clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c valid_age.cpp -o _objs/valid_age.pic.o
clang++ -shared -o _objs/libvalid_age.so _objs/valid_age.pic.o -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c test_valid_age.cpp -o _objs/test_valid_age.pic.o
clang++ -o test_valid_age.out '-Wl,-rpath,./_objs' -L./_objs _objs/test_valid_age.pic.o -lvalid_age -pie -pthread -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
mull-cxx \
    --linker="/usr/local/bin/clang++" \
    --mutators="cxx_all" \
    --linker-flags="-Wl,-rpath,./_objs -L./_objs -Wl,-S -fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -lvalid_age -pthread -lstdc++ -lm" \
    --ide-reporter-show-killed \
    --reporters=IDE\
    --print-options \
    --debug \
    test_valid_age.out

Expected Behavior

The tool should mutate both valid_age.cpp and test_valid_age.cpp.

Current Behavior

The tool is only validating test_valid_age.cpp

[info] Killed mutants (4/6):
/home/mull_issue_code/test_valid_age.cpp:5:17: warning: Killed: Replaced call to a function with 42 [cxx_replace_scalar_call]
    int test1 = valid_age(25) == true;
                ^
/home/mull_issue_code/test_valid_age.cpp:5:31: warning: Killed: Replaced == with != [cxx_eq_to_ne]
    int test1 = valid_age(25) == true;
                              ^
/home/mull_issue_code/test_valid_age.cpp:11:17: warning: Killed: Replaced call to a function with 42 [cxx_replace_scalar_call]
    int test2 = valid_age(20) == false;
                ^
/home/mull_issue_code/test_valid_age.cpp:11:31: warning: Killed: Replaced == with != [cxx_eq_to_ne]
    int test2 = valid_age(20) == false;
                              ^
[info] Survived mutants (2/6):
/home/mull_issue_code/test_valid_age.cpp:5:9: warning: Survived: Replaced 'T a = b' with 'T a = 42' [cxx_init_const]
    int test1 = valid_age(25) == true;
        ^
/home/mull_issue_code/test_valid_age.cpp:11:9: warning: Survived: Replaced 'T a = b' with 'T a = 42' [cxx_init_const]
    int test2 = valid_age(20) == false;
        ^
[info] Mutation score: 66%

Versions

mull-cxx

Mull: LLVM-based mutation testing
https://github.com/mull-project/mull
Version: 0.11.2
Commit: a9a8ed6c
Date: 01 Oct 2021
LLVM: 11.0.1

[AlexDenisov]

Hi @duthades, thank you for reporting the issue!

Indeed, mull should be mutating the shared object, but it's not supported in an automated way at the moment.
As a workaround, I can recommend using -mutate-only flag against the shared object, i.e. you need to mutate both the executable and the shared object separately, and then run the tests manually using mull-runner.

Also, more details are here (see the "two-step process).

[duthades]

Hi @AlexDenisov, I tried to mutate both executable and the shared object separately and then run the test manually using mull-runner but, I am still seeing the executable mutants. Here are the commands I used:

clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c valid_age.cpp -o _objs/valid_age.pic.o
clang++ -shared -o  _objs/libvalid_age.so _objs/valid_age.pic.o -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
mull-cxx --disable-junk-detection --debug --linker="/usr/local/bin/clang++" --mutators="cxx_all" --linker-flags="-shared -fPIC -pipe -fuse-ld=lld -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state" --mutate-only -output=_mutated_objs/libvalid_age.so  _objs/libvalid_age.so 

clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c test_valid_age.cpp -o _objs/test_valid_age.pic.o
clang++ -o test_valid_age.out '-Wl,-rpath,./_mutated_objs' -L./_mutated_objs _objs/test_valid_age.pic.o -lvalid_age -pie -pthread -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
mull-cxx --disable-junk-detection --debug --linker="/usr/local/bin/clang++" --mutators="cxx_all" --linker-flags="-Wl,-rpath,./_mutated_objs -L./_mutated_objs -Wl,-S -fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -lvalid_age -pthread -lstdc++ -lm" -mutate-only -output=test_valid_age_mutated.out test_valid_age.out

mull-runner -ide-reporter-show-killed test_valid_age_mutated.out

Am I missing something here?

[duthades]

Hi @AlexDenisov, did you have a chance to look at this issue?

[AlexDenisov]

Hi @duthades,

After some thinking, I know what's the problem is!
mull-cxx embeds the information about the mutants into the binary itself. In your case, you get the information both in mutated_objs/libvalid_age.so and in the test_valid_age_mutated.out.
Then, mull-runner extracts the information about the mutants from the executable under test.
This means that if the executable has some mutants, mull-runner will only run those mutants (as it didn't extract the info about mutants from the shared library).

The best solution would be to teach mull-runner to also scan all the dependent shared libraries, but it might be a demanding task performance-wise.

As a workaround, I can suggest looking at this tutorial https://mull.readthedocs.io/en/latest/tutorials/NonStandardTestSuite.html.
You may consider the test suite to be a "non-standard" and call mull-runner this way:

mull-runner -ide-reporter-show-killed mutated_objs/libvalid_age.so -test-program=test_valid_age_mutated.out

This way, it will search for mutants at mutated_objs/libvalid_age.so, but will run test_valid_age_mutated.out as a test suite.
The drawback, is that you cannot get mutants for both test_valid_age_mutated.out and mutated_objs/libvalid_age.so in one run. But, as a workaround to that, you can run the tests twice 😄

This is not ideal, but it should do the job.

[duthades]

Hi @AlexDenisov, This workaround is working on the above example code. I was wondering if there is a timeline to implement a feature to automatically scan the dependent shared files for mutations?

This workaround has made the integration of this tool in our pipeline very complex. Here is the psedocode for integeration:

def mutate(target):
    all_mutations = []	
    all_shared_libs = get_all_shared_files_for(target)
    for each_lib in all_shared_libs:
        linker_flags = get_linker_flags(each_lib)
        compiler_flags = get_compiler_flags(each_lib)
        mutate_shared_lib(each_lib, linker_flags, compiler_flags)
        mutated_lib = get_mutated_lib_for(each_lib)
        all_mutations.append(check_mutation_status(mutated_lib, target))
    return all_mutations

Finding linker flags and compiler flags is not always straight forward for dependent shared files (we use bazel for building and testing). This way the runtime is also increases by len(all_shared_lib) times.

[AlexDenisov]

Hi @duthades, thank you so much for trying and confirming that the workaround actually works!

I can certainly see the pain with this solution.
I cannot provide an ETA for the proper fix, but this is certainly an important problem to solve - I simply overlooked this use case when I initially implemented it.

I'll see how much effort it'd take and will get back to you soon.

[duthades]

Hi @AlexDenisov, I have noticed one more issue while testing. When I modify valid_age.cpp to use static variable. For example:

#include "valid_age.h"

int valid_age(int age) {
  static int lower_limit = 21;
  if (age >= lower_limit) {
        return 1;
    }
    return 0;
}

When I try to mutate this shared object file using the following commands:

clang++ -fPIC -g0 -Wall '-Wno-error=self-assign-overloaded' -fembed-bitcode -g '-std=c++14' -no-canonical-prefixes -Wno-builtin-macro-redefined -c valid_age.cpp -o _objs/valid_age.pic.o
clang++ -shared -o  _objs/libvalid_age.so _objs/valid_age.pic.o -pipe '-fuse-ld=lld' -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state
mull-cxx --disable-junk-detection --debug --linker="/usr/local/bin/clang++" --mutators="cxx_all" --linker-flags="-shared -fPIC -pipe -fuse-ld=lld -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state" --mutate-only -output=_mutated_objs/libvalid_age.so  _objs/libvalid_age.so 

Mull throws following error:

...
[info] Link mutated program (threads: 1)
[error] Cannot link program
status: Failed
time: 36ms
exit: 1
command: /usr/local/bin/clang++ -shared -fPIC -pipe -fuse-ld=lld -Wl,-z,relro,-z,now,-z,notext -no-canonical-prefixes -pthread -Wl,--push-state -Wl,-as-needed -ldl -lrt -Wl,--pop-state /tmp/mull-8ef324.o -o _mutated_objs/libvalid_age.so
stdout: 
stderr: ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(cxx_assign_const:/home/mull_issue_code/valid_age.cpp:6:9)

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(cxx_assign_const:/home/mull_issue_code/valid_age.cpp:8:5)

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(cxx_ge_to_gt:/home/mull_issue_code/valid_age.cpp:5:13)

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(cxx_ge_to_lt:/home/mull_issue_code/valid_age.cpp:5:13)

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(cxx_init_const:/home/mull_issue_code/valid_age.cpp:6:9)

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(cxx_init_const:/home/mull_issue_code/valid_age.cpp:8:5)

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(cxx_remove_void_call:/home/mull_issue_code/valid_age.cpp:3:19)

ld.lld: error: relocation R_X86_64_32S cannot be used against local symbol; recompile with -fPIC
>>> defined in /tmp/mull-8ef324.o
>>> referenced by valid_age.cpp:5
>>>               /tmp/mull-8ef324.o:(mull__Z9valid_agei_original)
clang++: error: linker command failed with exit code 1 (use -v to see invocation)


[error] Error messages are treated as fatal errors. Exiting now.

[duthades]

Hi @duthades, thank you so much for trying and confirming that the workaround actually works!

I can certainly see the pain with this solution. I cannot provide an ETA for the proper fix, but this is certainly an important problem to solve - I simply overlooked this user case when I initially implemented it.

I'll see how much effort it'd take and will get back to you soon.

@AlexDenisov did you have any chance to estimate effort? And probably an rough ETA for the fix?

[AlexDenisov]

@duthades the issue is that Mull doesn't respect certain compiler flags when lowering bitcode to the machine code.
#929 contains a fix that you can test by installing the "right" version (0.13.0-pr929 in this case).
Let me know if it works for you, then I'll cut the release soon.

UPD: here is the list of the versions: https://cloudsmith.io/~mull-project/repos/mull-nightly/packages/?q=version%3A0.13.0-pr929

Also, it's best to ensure that Mull uses the same major version of Clang/LLVM you use to build the software.

[duthades]

Hi @AlexDenisov, I compiled and installed mull with this PR changes for llvm 11. This change did not show mutated shared libs. I used the code and command from this comment - #903 (comment) with --lower-bitcode flag and without. Am I missing anything?

[AlexDenisov]

@duthades my latest comment only addresses the issue with the -fPIC, the original issue with the missing mutants from shared libs is not yet fixed.
Or did I misunderstand your message?

[duthades]

@AlexDenisov I got it now. I misunderstood your comment earlier. I tried to run the new mull tool using the code with static variable and commands from #903 (comment). The tool was able to mutate successfully now. :)

[AlexDenisov]

@duthades #931 is supposed to fix the original issue. Please, feel free to reopen the issue if the issue is still there.

Regarding the rest (specifically #903 (comment)), we are working on a clang plugin that would allow more seamless build system integration (at least we hope so).