Bench-Bot v2.0 Specification

[shawntabrizi]

The bench bot was originally created to specifically be used for the node-bench utility. I have, over the last month or so, hacked on top of it the ability to also be used for the runtime benchmarks process.

But these hacks have mostly been... hacky, and some better design and architecture could be used to make the whole tool more robust, easy to upgrade, and more usable.

I suggest the following specification for how users should interact with the bot:

/benchmark (interchangeable with `/bench`)
or
/benchmark help

Return a list of available commands as a Github comment.

/benchmark status

Return availability status and other information about the running processes and queue.

/benchmark specs

Return the machine specifications of the benchmarking device.

/benchmark node <benchmark_filters>

Run a node benchmark using the benchmark filters.

Only whitelisted users should be able to trigger this command.

/benchmark runtime <runtime> <pallet>

Run a runtime benchmark for a specific runtime using the default flags.

runtime options: Substrate, Polkadot, Kusama, Westend, Template(?)

pallet options: Either a specific pallet name or all, which should do some magic to query the available benchmarks on the machine and run them all.

Only whitelisted users should be able to trigger this command.

/benchmark runtime custom <flags>

Run a benchmark with a specific set of flags (thus no need to also specify the runtime).

Only whitelisted users should be able to trigger this command.

/benchmark cancel

Cancel a running benchmark.

Only if you are the person who initiated the benchmark or you are on some whitelist (like Parity Tech Core Devs)

/benchmark cancel all

Clear the entire queue. Only whitelisted users should be able to trigger this command.

/benchmark update

Ideally, update the running process using the latest on the master branch of this repo and restart the bot.