Support for testing

[PavelVozenilek]

V language may support thorough and exploratory testing.

1] Tests could be written like this:

TEST() // no need to invent a name
{
  assert(2 + 2 == 4);
}

TEST()
{
  assert(2 + 2 == 5);
}

(I was able to to implement such terse syntax in C++, C and Nim.)

Adding new empty test is then matter of few seconds, there is no need to register anything, to import a test library, to invent a new name, nothing at all.

2] The tests could be optionally compiled in (or omitted) in both debug and release modes. Testing in the release mode is a quite useful insurance against Heisenbugs.

3] Tests should be invoked manually (not automagically, as in D).

E.g.

 fn main()
 {
     if (tests-compiled-in) { // compiler defined constant or something
        run_tests(ONLY_RECENT_ONES);
     }
     ....
 }

For example, one may like to invoke only tests in source files modified recently (within last few hours). Then it may be practical to make run of recent tests every time whenever application starts.

4] A superfeature would be ability to replace (mock) existing functions inside given test, e.g.

TEST() 
{
   // inside this test the provided implementation of fopen() would be used
   replace fn fopen(...) { ... } 
   ... // code doing fopen(), uses the definition above
}

Having such feature would eliminate the architecture changes, mocking frameworks, dependence injections and other complicated mechanisms.

Internally, this could be implemented using functions pointers. Call to fopen() would always go through the function pointers. Normally it would end in the standard fopen(), but inside the test the value of function pointer would change to the new definition, and then back.

5] Tests could have optional attributes, whatever the user needs:

TEST(max-time <  300 ms) // if the test takes more time, show error
{
  ...
}

TEST(used-to-collect-performance-data)
{
  ...
}

TEST(shows-GUI, max-time < 20 ms)
{
  ...
}

Ideally, one would use own attributes freely, and then, in main function, one could go through these tests, check the attributes and invoke those tests desirable in the moment.

Running tests useful for performance collecting tools is one use case (tests checking edge cases should not mess with performance data).

6] The test could check that there is no memory leak inside. I have implemented this, and it really helps to find leaks early.

7] Depending how assert() is implemented: if it is debug mode only feature, then one may add variant, say verify(), which is valid also in release mode, when test are compiled in, and valid inside these tests only.

fn foo(...)
{
  assert(x < 0); // only in debug mode
  ...
}

TEST()
{
  foo(...);
  verify (y == true); // checks even in release mode
}

fn bar(...)
{
  verify(...); // not allowed here
}

8] If such functionality is available, then these tests can be seen as small, independent programs, useful for exploration and gaining insight. When one inherits large code base, such tests could be a way to familiarize with it, step by step.

[andrenth]

Number 4 is really intriguing, but wouldn't it require an extra pointer indirection for every function call? Unless you declare some specific functions as "replaceable". Even then, it would be nice if this overhead would exist only during test runs.

[PavelVozenilek]

@andrenth: yes, those function would be accessed indirectly, but only when tests are compiled in. Which is not expected be too performance sensitive.

The compiler may generate two sets of replaced functions and all functions which use them. One set (using indirections) would be called by the tests, the other set (fast one) by non-test code. This however feels as an overkill.

There may be a very special situation, where some tests are dedicated to performance testing or performance regression checks. This situation could be handled as separate case, employing compiler switch, and in this case this mock feature may not be available (compiler would issue an error).

[PavelVozenilek]

Another possibly useful feature, especially for the generics. Tests which intentionally do not compile.

Sometimes it may be desirable to check that certain construct doesn't work. Possible syntax:

TEST(does-not-compile)
{
   ...
}

The compiler would understand it as a special case, would check the code only for correct syntax (e.g. no unbalanced brackets), and then verify it doesn't compile.

It should be impossible to invoke such a test manually. No other attributes should be allowed together with the does-not-compile , nor any mocks inside.

[medvednikov]

This is very interesting, thanks!

Right now V has a very simple testing system:

fn test_foo() {
  assert foo(10) == 20
}

[PavelVozenilek]

Here's an idea of feature integrated into the testing framework.

The Problem

It is easy to check returned values for a function.

TEST()
{
  assert(add(2, 2) == 4);
}

However, one may want to be sure of other, non-obvious things:

• there's exactly one heap allocation going inside the add, it takes 41 bytes exactly, and it returns them back to the heap
• the function doesn't do any file I/O
• it tries to connect to this specific IP address, it fails, then it tries second time, and also fails. No more attempts.

This usually solved by ad-hoc stepping through in debugger, or worse, by making complex architectural designs to accomplish these checks. Dependency injection comes to my mind.

---

Simpler Solution
It is to use logging, designed for performance and unobtrusiveness.

I was inspired by this article:
http://akkartik.name/post/tracing-tests
and made a C library. It looked like:

void* my_malloc(uint sz) // replacement for malloc()
{
  ...
  TRACE("malloc %u bytes", sz);
  return p;
}

...

TEST()
{
  record_traces("malloc", "fopen", "this", "that"); // here I say which traces I want to collect

  assert(add(2, 2) == 4);

  // check the expected heap allocation
  const char* s = find_trace("malloc");
  if (!s) assert(false);
  ... extract bytes count from the string, expecting it to look like "malloc 41 bytes"
  assert(bytes == 41);

  // check no file I/O happened
  s = find_trace("fopen");
  assert(!s);
}

There were 4 separate phases:

1. In the beginning of a test I express interest which traces to collect, by specifying string prefixes. (This to avoid collecting everything.)
2. I run the code which collects the traces.
3. I use functions which go through the collected traces and find desired traces. I may extract more details from these traces. The result either confirms my expectations, or problem is asserted.
4. When the test ends, it clears up all collected traces, to make the desk clean for the next test.

It worked, but it was not very performant and it cluttered source code way too much.

My conclusion was: language support is needed to make this tool usable.

---

What can the language do?

1] Some traces could be inserted automagically by the compiler.

Instead of manually adding traces that a function has been invoked:

    void* my_malloc(uint sz) {
      TRACE("fn malloc");
       ...
    }
    void my_free(void* p) {
      TRACE("fn free");
       ...
    }

The compiler would take a note which function-call-traces are requested inside the tests and inserts them automagically. It would then look like:

    TEST()
    {
      record_traces("fn free", fn malloc");
      ...
    }

    void* my_malloc(uint sz) {
       ... // trace secretly added by the compiler here
    }
    void my_free(void* p) {
       ... // trace secretly added by the compiler here
    }

If the simple errors proposal (#3 (comment)) is implemented, returning an error could be also traced automatically.

TEST()
{
  record_trace("err.SomeKindOfError");
  ...
}

fn foo()
{
   ...
   if someting { 
      // <<<=== TRACE("err.SomeKindOfError in fn foo") will be inserted here by the compiler
      return err.SomeKindOfError; 
   }
}

Since the compiler exactly know which traces are requested, it would insert only those necessary traces. Not only the code would look cleaner, but functions/errors/etc which are not interesting won't be touched. Performance win.

---

2] The compiler should be able to check validity of trace strings.

void foo()
{
  TRACE("foo did X");
}

TEST()
{
  register_traces("foo did Y"); // typo here
  ...
  find_trace("foo did Z)"; // another typo
}

Above, the compiler should be able to notice that requested trace is never ever recorded. This is clearly typo or something was forgotten.

Similarly, within a test the traces should be consistent:

TEST()
{
  record_traces("A", "B", "C");
  ...
  find_trace("A"); // OK
  find_trace("b"); // not recorded even if it exists, clear mistake
}

Duplicated traces are probably also mistake:

fn foo()
{
  TRACE("foo");
   ...
}
fn bar()
{
  TRACE("foo"); // <<<=== probably an error
   ...
}

---

3] Without support by the compiler tracing is slow.

In my library:

void foo()
{
  TRACE("XYZ")
}

the TRACE did the following:

foreach traces-to-be-recorded -> a-trace
{
  if (strncmp(a-trace, "XYZ", strlen(a-trace)) == 0) {
   store_trace("XYZ");
  }
}

While a typical test asked to record only few traces, every time there was a TRACE in the code, it had to do this string lookup.

Compiler supported tracing may look internally like:

if (global_flag_for_traces_enabled_right_now && specific_trace_flag_829873) {
  store_trace("XYZ");
}

The global flag would be set only in tests which deal with tracing, and for every recorded trace there would be unique flag created by the compiler, flipped on only inside relevant test.

Traces which are never used (e.g. in libraries) would be always turned into no-op.

---

4] More performance improvements:

• if tests are not compiled in, all traces would be no-op
• if there is a special mode for tests doing benchmarking/performance data collection/etc, traces would be no-op. The compiler would ensure that these tests do not try tracing.
• fixed string traces could be stored as pointers, not full strings

---

5] Very advanced compiler support.

Sometimes I had to write code like:

void foo()
{
#ifdef TRACING // ugly, distracting, takes space
  int total = x + y;
  TRACE("total = %d", total);
#endif  
}

I dream about the language which would know that if traces are not compiled in, related code would be neither:

void foo()
{
  int total = x + y; // automagically no-op if tracing is not compiled in
  TRACE("total = %d", total);
}

Advanced IDE could highlight this situation.

---

6] What is not needed.

The article that inspired me intended to record everything during regular application run. Then user would be able to browse the logs and discover what was going on.

While this browsing may be useful when there's no debugger, it is unfeasible for non-trivial projects, and it would also overload the user.

Also, trace mechanism should not be seen as alternative or replacement for logging system, if it is used. Traces are obsessed with minutae details, low level information unusable for non-developers.

---

The main advantage of test tracing is that people won't be tempted to invent complicated things to accomplish equivalent functionality.

If they are not interested in this functionality, they won't be affected, except for some TRACE in libraries, and these could be easily ignored. Very advanced IDE may even hide them.

[PavelVozenilek]

The tests should have full access to private items and should be able to modify read only values.

They should be also able to temporarily modify constants (e.g. some timeout value), with compiler resetting the value back automatically at the end of the test.

[PavelVozenilek]

Few more desirable features:

---

[1] Tests placed inside the code, in rare cases and for really simple tests. It may look like:

if ... {
   ...

   TEST() { ... } // simple test testing this branch
} else {
   ...

   TEST() { ... } // simple test testing this branch
}

It should be used only sparingly.

---

[2] Feature from Zig language. Many tests share common setup and teardown code. This duplication can be avoided:

TEST() 
{
  common-setup-code

  TEST()
  {
    testA
  }

  TEST()
  {
    testB
  }

  common-teardown-code
}

This would be transformed into:

TEST() 
{
  common-setup-code
  testA
  common-teardown-code
}

TEST() 
{
  common-setup-code
  testB
  common-teardown-code
}

The goal to reduce lines used for testing is highly desirable in systems with thousands and thousands of tests.

---

[3] Each and every test could check for memory leaks. To accomplish this debug global variables (#208) would be needed.

It could also check for "leaked" threads and unlocked mutexes.

If one decides to use allocator of own design, the system should allow to reimplement the checking (e.g. by rewriting the test runner).

[nedpals]

Just wondering since this issue has been open for over a year now and there has been no movement on what features regarding testing that should be incorporated into the compiler. Should we reopen #3451 instead and discuss it there?

[nedpals]

Closed. Decided to reopen and continue the discussion on the recent #3451 instead.