Enable Running Zig Benchmarks in zBench Without Caching

[hendriknielaender]

Description

In the current setup of the zBench project, Zig tests are cached after their first successful execution. This behavior is standard for Zig, where tests are not re-run unless the tested code changes. However, for benchmarking purposes, we need the ability to run tests multiple times without relying on code modifications to trigger re-execution.

Issue

The caching mechanism in Zig's build system is efficient for many use cases but poses a challenge for continuous benchmarking. We need a method to run benchmarks in the zBench project repeatedly, irrespective of code changes, to obtain consistent and accurate performance measurements.

Current Workarounds

• Building and Running Binaries: One approach is to compile tests into binaries, allowing them to be executed repeatedly as needed. This method works but may not be ideal for all scenarios.
• Temporarily Disabling Caching: Another option is to redirect the build cache to a temporary directory, effectively disabling caching. However, this is a broad approach and may not be suitable for our specific needs.

Proposed Solution

A more refined approach is needed to handle benchmarks in Zig, especially for projects like zBench that require repeated test executions for accurate performance assessment. Some potential solutions could include:

• Benchmark Mode in Build System: Implementing a 'benchmark mode' in the build system that bypasses the caching mechanism for specific tests.
• Standard Library Support: In the long run, integrating benchmarking as a feature in the Zig standard library could provide a standardized and efficient way to handle such cases.

[FObersteiner]

so that this doesn't get lost: re-posting my review comment as I think this is relevant here.

Side note to having benchmarks as executables instead of tests: I've had some discussion about this on ziggit (https://ziggit.dev/t/exclude-test-from-being-cached/2275/, https://ziggit.dev/t/how-to-enable-more-logging-and-disable-caching-with-zig-build-test/2654), see also here on github. [...]

To summarize the github issue discussion: The language maintainers decided against benchmarks as part of the Zig testing framework. This might not be a totally final decision (Zig is still not v1.0), but I'd interpret it this way: at the moment, 3rd party benchmarking frameworks like zBench are the way to go.

With regard to caching; there are basically two options which seem ok to me:

• benchmark in a fn main(): build an executable, then run on demand
• benchmark in a test "...": set test.has_side_effects = true with test being the run artifact (std.Build.Step.Run) so that the test is ran each time (although only re-compiled if source changed)

[hendriknielaender]

Thank you for summarizing the discussion and the current stance of the Zig language maintainers.

Regarding the benchmarking implementation, I've given it some thought and experimentation. While the idea of using test "...": set test.has_side_effects = true is fascinating, I encountered some challenges in making it work as desired. After some trials, I've concluded that reverting to the benchmark in a fn main() approach seems to be the more straightforward and effective method for our needs at this time.

The main function approach allows us greater control and simplicity in building and running benchmarks. It's more aligned with the conventional execution flow, making it easier to manage and understand, especially for new contributors or when integrating with other tools.

I'm open to revisiting the has_side_effects method in the future, especially if there are updates or improvements in zig that make it more viable. For now, however, I believe focusing on the main function approach is the preferable path forward.

I'm looking forward to your thoughts and any further suggestions you might have on this matter.