Extension/Plugin Support

[hendriknielaender]

Proposal: Extension/Plugin Support in zBench

Overview

We aim to introduce a streamlined plugin support mechanism into zBench, facilitating easy creation of output format extensions like zbench-csv or zbench-json. This enhancement is designed to lower the barrier to extending zBench functionalities, allowing for a broader range of output formats tailored to different use cases and preferences.

Rationale

The current output capabilities of zBench, while comprehensive, are fixed and might not suit all users' needs. By enabling plugin support, we open up the door for customizations.

Proposal

The proposal outlines a simple yet flexible plugin architecture that allows plugins to be easily integrated with minimal changes to zBench's core codebase.

Plugin Interface

A single-function plugin interface that takes benchmark results and outputs them in the desired format. This function can be specified directly in the Config struct, allowing for high flexibility with minimal overhead.

Config Integration

Extend the Config struct to include an optional plugin function. This function, if provided, will be called with the benchmark results, and it is responsible for processing and outputting the data accordingly.

Implementation Example

const std = @import("std");

pub const PluginFn = fn (results: BenchmarkResults) void;

pub const Config = struct {
    ...
    output: ?PluginFn = null,
};

Usage

const zbench = @import("zbench");
const jsonPlugin = @import("zbench-json.zig");

var config = zbench.Config{
    ...
    output: jsonPlugin.outputResults,
};

zbench.runBenchmark(config);

[bens]

An issue with providing a function pointer is it would have to have its writer output fixed statically, I don't see how you could decide to write to a file path only known at run time, or switch between stdout and a file based on runtime information, or whatever else you might want to do. You could pass a *anyopaque along with it as a context value but that gets annoying and unsafe. I don't see a significant benefit over just calling a function with the Results to produce the output.

[hendriknielaender]

Leveraging function pointers indeed introduces limitations when we consider runtime decisions, such as toggling between stdout and file output or specifying file paths dynamically.

Instead of relying on function pointers, we could consider a design where plugins or extensions can define their output mechanisms more declaratively. This would allow for greater flexibility.

Maybe something like this:

pub const Plugin = struct {
    /// Sets the output target for the plugin. Could be a file path, stdout, etc.
    /// This method allows for runtime flexibility in determining the output destination.
    setOutputTarget: fn (self: *Plugin, target: OutputTarget) void,

    /// Processes the benchmark results. This method is where the plugin does its primary work,
    /// such as formatting results as CSV or JSON.
    processResults: fn (self: *Plugin, results: BenchmarkResults) void,
};

pub const OutputTarget = enum {
    stdout,
    file(path: []const u8),
};

[bens]

What's the benefit over calling a function with a BenchmarkResults? The output capabilities of that are unlimited and as flexible as anything can be because a BenchmarkResults contains all the relevant information which can be formatted in whatever way the user wants. Given that's already there, I don't see a need for this configuration parameter, it's only going to be unnecessary complexity.

Another example is sending the results to a network socket, or a buffer, the point isn't to add one more option to handle those cases, it's that the writer interface is intentionally open ended and it's perfectly well supported by a function with a writer: anytype parameter.

[hendriknielaender]

True this is a valid point regarding the inherent flexibility in using a function that accepts BenchmarkResults and a writer of anytype.

The proposal for a plugin interface aims not to restrict this flexibility but to complement it by offering a structured way to manage multiple output formats and destinations. I think the plugin approach can add value for these points:

• Standardization: By defining a common interface for plugins, we establish a standard method for extending zBench.

• Ease of Use: For users not familiar with Zig's generics or those looking for quick, out-of-the-box solutions, plugins provide a simple and discoverable way to extend functionality. It's about offering choice and convenience.

• Encapsulation and Composition: Plugins allow for encapsulating functionality related to output formatting and destination management.

• Lifecycle Management: A plugin system can manage the lifecycle of extensions, including initialization, configuration, execution, and teardown. This is particularly useful for resources that require setup or cleanup, like network connections or file handles.

[bens]

Fair enough, some prototype code to play with might help to clarify how it would work.