Migrate to std.time.Timer and tweak benchmark boundaries

[FObersteiner]

This will address #17 and #18.

• add a simple benchmark example that allows to easily test different boundaries, e.g. maximum number of runs and maximum duration
• revise above mentioned parameters to get a stable setup for testing on different platforms
• implement std.time.Timer based version of zBench (basically clean-up of the draft in branch std-Timer)
• collect some data
• maybe: find a way to have an integration test based on the sleep example

[FObersteiner]

        // Print the progress if it's a new percentage
        const currentProgress: u8 = @truncate(progress);
        if (currentProgress != lastProgress) {

after making this print stuff, I realize that this might clutter the console - made me think if we should introduce a verbose flag :)

[hendriknielaender]

        // Print the progress if it's a new percentage
        const currentProgress: u8 = @truncate(progress);
        if (currentProgress != lastProgress) {

after making this print stuff, I realize that this might clutter the console - made me think if we should introduce a verbose flag :)

Yeah right! 👍 good point

[FObersteiner]

Results on Windows look good to me now. I wrote a small Python script to repeatedly call the benchmark and average the results for me. See below.

Windows

std.time.nanoTimestamp

The "normal" Windows clock is obviously not suitable for our purpose. To make this work at all, I had to add an additional bounds-check that the recorded durations aren't all-zero.

   D:\Code\Zig\zBench   main ●  ?1 ~1   3.11.4  zig build test_examples -Doptimize=ReleaseSafe
   D:\Code\Zig\zBench   main ●  ?1 ~1   3.11.4  python .\repeat_bench.py basic
My Benchmark         463.0ns      (0.0ns ... 10.50ms) 0.0ns      0.0ns      0.0ns
My Benchmark         278.0ns      (0.0ns ... 8.8ms) 0.0ns      0.0ns      0.0ns
My Benchmark         461.0ns      (0.0ns ... 10.2ms) 0.0ns      0.0ns      0.0ns
My Benchmark         462.0ns      (0.0ns ... 10.0ms) 0.0ns      0.0ns      0.0ns
My Benchmark         465.0ns      (0.0ns ... 10.40ms) 0.0ns      0.0ns      0.0ns
My Benchmark         312.0ns      (0.0ns ... 10.49ms) 0.0ns      0.0ns      0.0ns
My Benchmark         305.0ns      (0.0ns ... 10.1ms) 0.0ns      0.0ns      0.0ns
My Benchmark         304.0ns      (0.0ns ... 9.996ms) 0.0ns      0.0ns      0.0ns
My Benchmark         312.0ns      (0.0ns ... 10.44ms) 0.0ns      0.0ns      0.0ns
My Benchmark         465.0ns      (0.0ns ... 9.948ms) 0.0ns      0.0ns      0.0ns
--- mean: 382.7ns, avg.min: 0.0ns, avg.max: 10087400.0ns ---

std.time.Timer

While std.time.Timer works, you can still observe Windows' clock granularity of 100 ns in the minimum duration and the percentiles:

   D:\Code\Zig\zBench   dev ●  ?1   3.11.4  zig build test_examples -Doptimize=ReleaseSafe
   D:\Code\Zig\zBench   dev ●  ?1   3.11.4  python .\repeat_bench.py basic
My Benchmark         393.0ns      (300.0ns ... 30.0µs) 400.0ns    400.0ns    400.0ns
My Benchmark         394.0ns      (300.0ns ... 19.100µs) 400.0ns    400.0ns    400.0ns
My Benchmark         398.0ns      (300.0ns ... 27.400µs) 400.0ns    500.0ns    500.0ns
My Benchmark         395.0ns      (300.0ns ... 29.400µs) 400.0ns    400.0ns    400.0ns
My Benchmark         396.0ns      (300.0ns ... 18.900µs) 400.0ns    400.0ns    500.0ns
My Benchmark         402.0ns      (300.0ns ... 31.600µs) 400.0ns    600.0ns    700.0ns
My Benchmark         395.0ns      (300.0ns ... 18.500µs) 400.0ns    400.0ns    400.0ns
My Benchmark         394.0ns      (300.0ns ... 36.200µs) 400.0ns    400.0ns    400.0ns
My Benchmark         394.0ns      (300.0ns ... 20.400µs) 400.0ns    400.0ns    400.0ns
My Benchmark         397.0ns      (300.0ns ... 77.600µs) 400.0ns    400.0ns    500.0ns
--- mean: 395.8ns, avg.min: 300.0ns, avg.max: 30910.0ns ---

Linux

I was happy on Linux before, so I'm not going to repeat that test with the old version.

std.time.TImer

Also looking very good:

 I  %  ~/C/Z/zBench_fork   dev *+26/-42   zig build test_examples -Doptimize=ReleaseSafe
 I  %  ~/C/Z/zBench_fork   dev …  python repeat_bench.py basic      1410ms  Tue 21 Nov 2023 09:09:45 PM CET
My Benchmark         397.0ns      (393.0ns ... 10.254µs) 397.0ns    399.0ns    399.0ns   
My Benchmark         395.0ns      (391.0ns ... 11.969µs) 395.0ns    397.0ns    397.0ns   
My Benchmark         395.0ns      (391.0ns ... 11.871µs) 395.0ns    397.0ns    397.0ns   
My Benchmark         395.0ns      (391.0ns ... 10.158µs) 395.0ns    397.0ns    397.0ns   
My Benchmark         395.0ns      (391.0ns ... 10.698µs) 395.0ns    397.0ns    397.0ns   
My Benchmark         395.0ns      (391.0ns ... 14.176µs) 395.0ns    397.0ns    397.0ns   
My Benchmark         396.0ns      (393.0ns ... 11.783µs) 397.0ns    399.0ns    399.0ns   
My Benchmark         395.0ns      (391.0ns ... 10.108µs) 395.0ns    397.0ns    397.0ns   
My Benchmark         395.0ns      (391.0ns ... 11.957µs) 395.0ns    397.0ns    397.0ns   
My Benchmark         395.0ns      (391.0ns ... 10.851µs) 395.0ns    397.0ns    397.0ns   
--- mean: 395.3ns, avg.min: 391.4ns, avg.max: 11382.5ns ---

[hendriknielaender]

That looks good! 🥇
Maybe we need a more extensive benchmark besides the two basic examples, to maybe get a more meaningful result, what do you think?
For now this is fine.

[hendriknielaender]

What is the reason for MAX_N and MAX_ITERATIONS to be so specific?

[FObersteiner]

Those are powers of 2, no specific reason, sorry for the confusion ;-) The numbers are basically orders of magnitude where I found the benchmarks to run smoothly, and to not show too much variance.
I was thinking if we might give them reasonable defaults but expose as configurable, to find optimal settings for a specific scenario?

[FObersteiner]

That looks good! 🥇 Maybe we need a more extensive benchmark besides the two basic examples, to maybe get a more meaningful result, what do you think? For now this is fine.

jup, let's leave it at that for now, and save more elaborate examples / integration tests for later.