How can I mix production code with the Unit Tests?

[evandrocoan]

How can I mix production code with the Unit Tests?

Initial page says:

This allows the framework to be used in more ways than any other - tests can be written directly in the production code!

• This makes the barrier for writing tests much lower - you don't have to: 1. make a separate source file 2. include a bunch of stuff in it 3. add it to the build system and 4. add it to source control - You can just write the tests for a class or a piece of functionality at the bottom of its source file - or even header file!
• Tests in the production code can be thought of as documentation or up-to-date comments - showing how an API is used
• Testing internals that are not exposed through the public API and headers becomes easier!

How can I mix production code with the Unit Tests?

I am thinking about just doing the #include "doctest.h", and writing the tests down, but how the Unit Tests code will not end up on the production binary after compiled from the main program, instead of the Unit Tests Driver Class:

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

I found out this macro DOCTEST_CONFIG_DISABLE. Therefore how would it work, I would have to define it on every production source code file, to disable the Unit Tests?

[onqtam]

If you are using CMake you could add DOCTEST_CONFIG_DISALBE globally for a configuration like this:

set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -DDOCTEST_CONFIG_DISABLE")

The same can be done in any (meta) build system.

You could also create a "production" config which is like "release" (by copying those flags) but with the addition of this configuration identifier.

If you are using directly Visual Studio you could add the define for a particular config globally. Same for all other IDEs.

Does this help?

[evandrocoan]

Yes. For now I am only using a simple make file I wrote by hand. But I am looking for the available tools for Makefile generation.

I think I found the solution, to include the DOCTEST_CONFIG_DISABLE everywhere. All my source files include a debug tools header called debug.h which is controlled by the variable DEBUG_LEVEL defined on the top of it. So, inside it I put:

#if !( DEBUG_LEVEL & DEBUG_LEVEL_RUN_UNIT_TESTS )

    #define DOCTEST_CONFIG_DISABLE

#endif

#include "doctest.h"

So, I am not including the header doctest.h directly on any source code file, but only the header debug.h, which includes it and controls whether the Unit Tests are enabled or not.

Now all my project source code files are like these:

doctest_main.cpp

/**
 * This tells to provide a main() - Only do this in one cpp file.
 */
#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

doctest_tests.cpp

#include "project_sources_with_tests.cpp"
//...

project_sources_with_tests.cpp

#include "debug.h"

unsigned int factorial( unsigned int number )
{
    return number <= 1 ? number : factorial(number-1)*number;
}

TEST_CASE("testing the factorial function")
{
    CHECK(factorial(1) == 1);
    CHECK(factorial(2) == 2);
    CHECK(factorial(3) == 6);
    CHECK(factorial(10) == 3628800);
}

debug.h

/**
 * This is to view internal program data while execution. Default value: 0
 *
 *  0  - Disables this feature.
 *  1  - Basic debugging.
 *  2  - Run the `doctest` Unit Tests.
 */
#define DEBUG_LEVEL 1



#define DEBUG_LEVEL_DISABLED_DEBUG 0
#define DEBUG_LEVEL_BASIC_DEBUG    1
#define DEBUG_LEVEL_RUN_UNIT_TESTS 2

#if DEBUG_LEVEL > DEBUG_LEVEL_DISABLED_DEBUG

    #define DEBUG

    /**
     * Disables the `doctest` Unit Tests from being included/compiled to the binary output file.
     *
     * See:
     * https://github.com/onqtam/doctest/blob/master/doc/markdown/configuration.md
     */
    #if !( DEBUG_LEVEL & DEBUG_LEVEL_RUN_UNIT_TESTS )
        #define DOCTEST_CONFIG_DISABLE
    #endif

    #include "doctest.h"

#endif

This is the Makefile

PROGRAM_MAIN_DEBUGGER = debug

DOCTEST_TESTS_FILE   = doctest_tests
DOCTEST_DRIVER_CLASS = doctest_main

doctest_tests: $(DOCTEST_DRIVER_CLASS).o $(PROGRAM_MAIN_DEBUGGER).h
	g++ --std=c++11 $(DOCTEST_DRIVER_CLASS).o $(DOCTEST_TESTS_FILE).cpp -o main
	./main --force-colors=true

$(DOCTEST_DRIVER_CLASS).o: $(DOCTEST_DRIVER_CLASS).cpp
	g++ $(DOCTEST_DRIVER_CLASS).cpp -c -o $(DOCTEST_DRIVER_CLASS).o

[chambm]

I was about to ask a similar question. I made a multi-file test case here:
https://wandbox.org/permlink/qMyhHuckLLqsCfcB

The main program has two modes depending on whether "TEST" is defined. This is to emulate multiple binaries. Consider that I have a test executable which defines "TEST" and a production executable that doesn't. The doctest asserts are in class.cpp so that I can test anonymous-scoped functions. I'm trying to emulate the situation I have with my real code, but perhaps wandbox is oversimplifying it...

I am trying to add doctest tests to an existing codebase with 100s of test files. We use Boost.Buildwith custom test macros rather than Boost.Test. My current build system would only build class.cpp once even though it's used by both the test build and the production binary. The doctest asserts cannot go in the test files of course, they have to go in the actual TUs that have the anonymous functions to be tested.

My original thought was to put the doctest runner in the individual test files (which are used to test the public APIs), but that will result in doctest asserts being run multiple times. According to the discussion above, I need to modify the build config so that these TUs get built twice - once for the production code and once for the test code. Most of my TUs probably won't have doctest asserts, so this seems pretty painful. I guess I need to globally include and disable doctest, which can be overridden by a special build system function to create a testing bootstrapper for each TU I put doctests in. Is there a better way I'm missing?

[evandrocoan]

My original thought was to put the doctest runner in the individual test files

You should put the running on the Driver Class, which starts the Unit Tests execution, not on the main file where you program starts' running.

However you can put it there if you use a global variable to control whether the Unit Tests are enabled. For example, on my code I have a global include called debug.h as follows:

debug.h

/**
 * This is to view internal program data while execution. Default value: 0
 *
 *  0  - Disables this feature.
 *  1  - Basic debugging.
 *  2  - Run the `doctest` Unit Tests.
 */
#define DEBUG_LEVEL 1

#if DEBUG_LEVEL > 0

    #define DEBUG

    /**
     * Disables the `doctest` Unit Tests from being included/compiled to the binary output file.
     *
     * See:
     * https://github.com/onqtam/doctest/blob/master/doc/markdown/configuration.md
     */
    #if !( DEBUG_LEVEL & 2 )
        #define DOCTEST_CONFIG_DISABLE
    #endif

#endif

#include "doctest.h"

Now every file on my project includes this debug.h file. So, when I set the DEBUG_LEVEL to 2, it forces doctest to enable the Unit Tests across all my project files.

When I set the DEBUG_LEVEL to any other value than 2, doctest will exclude the unit tests from all my project files. Meaning they will be not included on the final binary file, which is like they did not existed across the project.

Now the problem is how to start the Unit Tests. This is done automatically by a driver class called doctest_driver_class.cpp, which contains only:

doctest_driver_class.cpp

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

So, when you build it you need to first build the driver class with:

g++ doctest_driver_class.cpp -c -o doctest_driver_class.o

And later compile/link together all the project files which include the Unit Tests. For example, let us assume do you have only one project file with Unit Tests and it is called source_file_with_tests.cpp:

source_file_with_tests.cpp

#include "debug.h"

unsigned int factorial( unsigned int number )
{
    return number <= 1 ? number : factorial(number-1)*number;
}

TEST_CASE("testing the factorial function")
{
    CHECK(factorial(1) == 1);
    CHECK(factorial(2) == 2);
    CHECK(factorial(3) == 6);
    CHECK(factorial(10) == 3628800);
}

Now to link together this project source file, you need to run the command:

g++ --std=c++11 doctest_driver_class.o source_file_with_tests.cpp -o main

And if you ran the main file, it will run you unit tests on the file source_file_with_tests.cpp.

If you have more files with unit tests like other_unit_test.cpp, you need to build like:

g++ --std=c++11 doctest_driver_class.o source_file_with_tests.cpp other_unit_test.cpp -o main

---

I guess I need to globally include and disable doctest, which can be overridden by a special build system function to create a testing bootstrapper for each TU I put doctests in.

I did not understand why to create a special bootstrapper for each TU you put doctest in. Perhaps the example I put above solves this problem?

[chambm]

Because I plan to have the doctests spread across my project hierarchy, it would be awkward to collect them all and run them once. Instead, I am compiling a second version of the TUs with doctests with a special flag, like your debug flag, to turn on doctest. Otherwise it will be disabled.

// without PWIZ_DOCTEST defined, disable doctest macros; when it is defined, doctest will be configured with main()
#ifndef PWIZ_DOCTEST
#define DOCTEST_CONFIG_DISABLE
#include "libraries/doctest.h"
#else
#define DOCTEST_CONFIG_IMPLEMENT
#include "libraries/doctest.h"
int main(int argc, char* argv[])
{
    TEST_PROLOG(argc, argv)

    try
    {
        doctest::Context context;
        testExitStatus = context.run();
    }
    catch (exception& e)
    {
        TEST_FAILED(e.what())
    }
    catch (...)
    {
        TEST_FAILED("Caught unknown exception.")
    }

    TEST_EPILOG
}
#endif

[onqtam]

A few notes on the wandbox example:

• by default it doesn't link because TEST is not defined and thus doctest isn't implemented anywhere, but is used in class.cpp.
• in class.cpp the commented out test that doesn't compile - the reason is that foo::wtf() is private - you will need to make it accessible somehow - perhaps by adding a static method to the class (which has access to wtf()) and then call it from a TEST_CASE()?

My current build system would only build class.cpp once even though it's used by both the test build and the production binary.

Do you have tests with your macros in class.cpp - because if you do then this would mean that in your current setup the tests are not stripped from the final binary shipped to customers?

My original thought was to put the doctest runner in the individual test files (which are used to test the public APIs), but that will result in doctest asserts being run multiple times.

there has to be only one runner in a binary (executable or shared object). If you implement the test runner in each test file then that would mean that each test file compiles to a separate binary. In that case the same assert wouldn't be run multiple times, but there will be different executables with different test registries...

---

when mixing tests and production code - the use of DOCTEST_CONFIG_DISABLE was intended and that would require a separate build...

When writing tests not in separate test files but in your production code files you could hack your build system to build only those TUs twice - with doctest enabled and disabled, but that would be a complicated setup. But it can be achieved.

@chambm I might be missing something when trying to understand your question/problem...

[chambm]

Yes hacking my build system is basically what I decided was the simplest setup. To make things more concrete, take a look at one of my Jamfiles:
https://sourceforge.net/p/proteowizard/code/HEAD/tree/trunk/pwiz/pwiz/utility/minimxml/Jamfile.jam

You can see that each class has its own unit test driver which gets compiled and run by Boost.Build (unit-test-if-exists, checks to make sure the file exists and won't fail if it doesn't, so we can deploy subsets of the source tree without tests). Suppose I wanted to add some doctests to an anonymous function in XMLWriter.cpp. Then I would add a new line:

doctest XMLWriterDoctest : XMLWriter.cpp pwiz_utility_minimxml ;

where doctest is defined as:

rule doctest ( name : sources + : requirements * )
{
    unit-test-if-exists $(name) : $(sources) : $(requirements) <define>PWIZ_DOCTEST <location-prefix>doctest ;
}

<define>PWIZ_DOCTEST enables the doctest driver function mentioned above, effectively turning the TU into a test file. location-prefix makes the TU compile in a different location so it doesn't interfere with the production build.

We have over 200 individual unit test files, so we use TeamCity as the overall test registry. Adding doctests to that registry is no problem (telling TeamCity about the tests is what TEST_PROLOG and TEST_EPILOG do).

[Trass3r]

Been thinking about this too. Building the code twice is out of the question in bigger projects which already suffer from insane compilation times thx to the lack of (proper) modules.

You could also create a "production" config which is like "release" (by copying those flags) but with the addition of this configuration identifier.

This is a good idea already. But that means in release builds we have at least the space overhead plus potential cache effects and all the test registration code running at startup every time even though it will never be used, which may or may not pose a problem idk (has anyone ever measured the overhead?).
Ideally that could be decided at link time: we have production and test code in the object files and produce 2 binaries from them. But I don't see how you could achieve that. Anyone got any clever ideas?
Edit: Maybe if you put the test cases into special sections and use a custom linker script... but then again the static initializer code still references them.