Proposal: companion file for the tests

[PavelVozenilek]

tl;dr:

When there's source file abc.zig (the master) then there could also be abc.zig-tests (test companion) file containing tests. The compiler will understand this arrangement.

---

There are few problems with test heavy code. It takes lot of effort, for example. Another trouble is: where to put all these tests?

Tests could be placed:

1. Just after the relevant function.
2. In a separate file.
3. At the end of the source file.

What are advantages and disadvantages of these schemes?

1. Advantages: it is the most natural placement. One can find relevant tests quickly. Disadvantages: source file grows a lot. It becomes harder to read such files, understand APIs they expose.
2. Advantages: source file is kept clean and small. Disadvantages: it is hard to find relevant tests, add new ones. Sometimes it is hard even to find the test file. Private things from the source file are not accessible in test file. Inflexibility, duplication. Test files need to be added into the build. Including these tests files in C (to have access to the privates) interferes with test support macros.
3. Advantages: almost none. Disadvantages: almost all.

---

My proposal:

1. Allow the choice, also allow combinations. This in practice means adding support for (2).
2. Whenever there's source file name.zig and also name.zig-tests, then this test companion is automatically compiled together with the master file.
3. Test companion file has full access to master file privates and shares all its imports.
4. Nothing can be exported or made public from the test companion file, it is only for the tests.
5. Master file cannot invoke anything from the test companion file. No other zig source can import it.

---

Optional feature is to allow easier orientation in the tests. There could be "test forwards":

////// abc.zig //////////
fn foo() { ... }

test "some basic foo test"
{
  ...
}

test "advanced foo test 1"; // two forwards to tests in abc.zig-tests
test "advanced foo test 2";

///// abc.zig-tests /////////////
test "advanced foo test 1"
{
   ...
}

test "advanced foo test 2"
{
  ...
}

Every "test forward" would need to have unique name (within the source file) and this name has to be present exactly once in test companion file.

Such feature would allow the reader to locate relevant tests and give a hint about test coverage.

---

This proposal can work together with the proposal for custom test runner #567.

[andrewrk]

have a look at std/special/compiler-rt/*

[PavelVozenilek]

@andrewrk: such tests do not have access to non-public functions/internal structures. It is easy to test purely functional math API, but how would it deal with a complex library?

[thejoshwolfe]

how would it deal with a complex library?

My suggestion: write functional tests (meaning tests that only use the public api) in a separate file, and try to get 100% coverage (#352). Then when you make a new version of the library, make sure your coverage hasn't regressed.

If there are parts of your library that are infeasible to get coverage on in this way, you could write private-access unit tests in the source file like what is already possible.

[PavelVozenilek]

100% coverage

Been there, done that. It is very exhausting and IMHO not worth the blood and tears.

The problem is that the most useful tests deal directly with internal parts (to set up edge cases). Moving these specific tests into separate file benefits readability most.

[PavelVozenilek]

An aside note.

Companion files could be also useful for other purposes. One of the "good" ideas I once got was to add internal HTTP server to the application, to show internal state of various program structures at any moment.

It didn't go well. It was virtually impossible to ensure that every displayed internal structure is made MT safe. Later I switched to generating HTML files for the whole app when demanded. This was much easier solution (no need to add MT safety everywhere) and mildly usable.

However, generating single HTML page could easily take thousands of LOC for nontrivial data structure. By definition the generator code needed full access to everything, so it could not be placed in a separate file.

If Zig has a way to define "this is how specific companion file for my project looks like" it would make such features less cumbersome.

[jido]

I believe some prog languages put test cases (or pre- and post-conditions) in the comment header of functions. This does not hurt readability of the actual code too much.

[PavelVozenilek]

@jido:

The use case I am thinking about has 50+% of code inside tests. This quickly becomes hard to read and manipulate. Especially tests of internal parts are long and hairy and would be better separated. However, this requires direct access to internals from that separate file.

Examples could be handled separately (and not in comments - hard to copy/paste): #507.