Golden Tests

Golden tests is a golden file testing library configured so that tests can be created and edited from the test files alone without ever touching the source code of your compiler, interpreter, or other tool.

Why golden tests?

Golden tests allow you to specify the output of some command within a file and automatically ensure that that output doesn't change. If it does, goldentests will show an error-diff showing the expected and actual output. This way, whenever the output of something changes a human can see the change and decide if it should be kept or is a bug and should be reverted.

What are golden tests useful for?

Golden tests are especially useful for applications that take a file as input and produce output of some kind. For example: compilers and config-parsers (well, parsers in general) are two such applications that can benefit from automated golden tests. In the case of a config parser, you would be able to provide many config examples as tests and ensure that your parser was able to read the files with the expected stdout/stderr output and exit code.

Example Output

Getting Started

As of version 1.1, there are now two ways to use goldentests - either as a standalone binary or as a rust integration test. If you want to run it as a binary, continue on. If not, skip ahead to the next section. With that out of the way, we can install goldentests via:

$ cargo install goldentests --features binary

An example usage looks like this:

$ goldentests /bin/python path-to-tests '# '

This will tell goldentests to run /bin/python on each file in the path-to-tests directory. You'll likely want to alias this command with your preferred arguments for easier testing. An example test for us may look like this:

print("Hello, World!")

# args: -b
# expected stdout:
# Hello, World!

This file tells goldentests to run the command /bin/python -b path-to-tests/example.py and issue an error if the output of the command is not "Hello, World!".

Note that there are test keywords args: and expected stdout: embedded in the comments. This is what the '# ' parameter was when we invoked goldentests. You can change this parameter to change the prefix that goldentests looks for when parsing a file. For most languages, this should be a comment of some kind. E.g. if we we're testing haskell, we would use -- as the test-line prefix.

As a rust integration test

The second way to use goldentests is as a rust library for writing integration tests. Using this method will have goldentests run each time you call cargo test. To get started plop this into your Cargo.toml:

goldentests = "1.1"

And create an integration test in tests/goldentests.rs. The specific name doesn't matter as long as the test can be picked up by cargo. A typical usage looks like this:

use goldentests::{ TestConfig, TestResult };

#[test]
fn run_golden_tests() -> TestResult<()> {
    let config = TestConfig::new("target/debug/my-binary", "my-test-path", "// ")?;
    config.run_tests()
}

This will tell goldentests to find all files recursively in my-test-path and run target/debug/my-binary to use the files in some way to produce the expected output. For example, if we're testing a compiler for a C-like language a test file for us may look like this:

puts("Hello, World!");

// args: --run
// expected stdout:
// Hello, World!

This will run the command target/debug/my-binary --run my-test-path/example.c and will issue an error if the output of the command is not "Hello, World!".

Note that there are test keywords args: and expected stdout: embedded in the comments. This is what the "// " parameter was in the rust example. You can change this parameter to change the prefix that goldentests looks for when parsing a file. For most languages, this should be a comment of some kind. E.g. if we we're testing haskell, we would use -- as the test-line prefix.

Advanced Usage

Here is the full set of keywords goldentests looks for in the file:

• args: <single-line-string>: Anything after this keyword will be used as the command-line arguments for the program that was specified when creating the TestConfig.
• expected stdout: <multi-line-string>: This keyword will continue reading characters, appending them to the expected stdout output until it reaches a line that does not start with the test prefix ("// " in the example above). If the stdout when running the program differs from the string given here, an appropriate error will be issued with a given diff. Defaults to "".
• expected stderr: <multi-line-string>: The same as expected stdout: but for the stderr stream. Also defaults to "".
• expected exit status: [i32]: If specified, goldentests will issue an error if the exit status differs to what is expected. Defaults to None (exit status is ignored by default).

You can even configure the specific keywords used if you want. For any further information, check out goldentest's documentation here.

Automatically updating tests

Optionally, tests can be automatically updated by passing the --overwrite flag when running goldentests as a standalone program, or by setting the overwrite_tests flag when running as a rust library. Doing this will update the expected output in each file so that it matches the actual output. Since this is all automatic, make sure to manually review any changes before using this flag.