benchee_dsl.livemd

BencheeDsl - A DSL for Benchee

Mix.install([
  {:benchee_dsl, "~> 0.5"}
])

Usage

BencheeDsl offers a DSL to write benchmarks for Benchee in an ExUnit style. For more informations to benchmarks and interpretation of the results see the Benchee documentation.

Note: Currently, this notebook does not run with the Livebook App.

Define a benchmark

A benchmark is a module that uses BenceeDsl.Benchmark. The macro job/2 defines the functions to benchmark.

defmodule Benchmark.One do
  use BencheeDsl.Benchmark

  @list Enum.to_list(1..10_000)

  defp map_fun(i), do: [i, i * i]

  job flat_map do
    Enum.flat_map(@list, &map_fun/1)
  end

  job map_flatten do
    @list |> Enum.map(&map_fun/1) |> List.flatten()
  end
end

Run benchmark

Benche runs the benchmark and writes the results to the console when the call 'Benchmark.One.run()' is made. See Benchee/Features for a description of the different statistical values and what they mean.

Benchmark.One.run()

With the option return: :result the function run/1 returns the Benchee.Suite struct with the results of the benchmark run.

Benchmark.One.run(return: :result)

Configuration

Benchee takes a wealth of configuration options, however those are entirely optional. Benchee ships with sensible defaults for all of these, see Benchee/Configuration. The configuration is passed as a keyword list with BencheeDsl.

Benchmark.One.run(warmup: 1, time: 3, print: [configuration: false])

The config macro can be used to directly write the configuration into the benchmark.

defmodule Benchmark.Two do
  use BencheeDsl.Benchmark

  config(warmup: 1, time: 3, print: [configuration: false])

  @list Enum.to_list(1..10_000)

  defp map_fun(i), do: [i, i * i]

  job flat_map do
    Enum.flat_map(@list, &map_fun/1)
  end

  job map_flatten do
    @list |> Enum.map(&map_fun/1) |> List.flatten()
  end
end

Benchmark.Two.run()

Metrics to measure

Benchee can't only measure execution time, but also memory consumption and reductions!

You can measure one of these metrics, or all at the same time. The choice is up to you. Warmup will only occur once though, the time for measuring the metrics are governed by time, memory_time and reduction_time configuration values respectively.

By default only execution time is measured, memory and reductions need to be opted in by specifying a non 0 time amount.

Benchmark.Two.run(memory_time: 2, reduction_time: 2)

Inputs

:inputs is a very useful configuration that allows you to run the same benchmarking jobs with different inputs. We call this combination a "scenario". You specify the inputs as either a map from name (String or atom) to the actual input value or a list of tuples where the first element in each tuple is the name and the second element in the tuple is the value.

Why do this? Functions can have different performance characteristics on differently shaped inputs - be that structure or input size. One of such cases is comparing tail-recursive and body-recursive implementations of map. More information in the repository with the benchmark and the blog post.

defmodule Benchmark.Three do
  use BencheeDsl.Benchmark

  defp map_fun(i), do: [i, i * i]

  job flat_map(input) do
    Enum.flat_map(input, &map_fun/1)
  end

  job map_flatten(input) do
    input |> Enum.map(&map_fun/1) |> List.flatten()
  end
end

inputs = %{
  "Small" => Enum.to_list(1..1_000),
  "Medium" => Enum.to_list(1..10_000),
  "Bigger" => Enum.to_list(1..100_000)
}

Benchmark.Three.run(inputs: inputs, time: 3, pre_check: true)

The inputs macro can be used to directly write the inputs into the benchmark.

defmodule Benchmark.Four do
  use BencheeDsl.Benchmark

  config(time: 3, pre_check: true, print: [configuration: false])

  inputs(%{
    "Small" => Enum.to_list(1..1_000),
    "Medium" => Enum.to_list(1..10_000),
    "Bigger" => Enum.to_list(1..100_000)
  })

  defp map_fun(i), do: [i, i * i]

  job flat_map(input) do
    Enum.flat_map(input, &map_fun/1)
  end

  job map_flatten(input) do
    input |> Enum.map(&map_fun/1) |> List.flatten()
  end
end

Benchmark.Four.run()

Capture a job

The macro job/1 accepts also a capture (&/1) as argument. The benchmarked functions are written in a separate module.

defmodule Benchmark.Jobs do
  defp map_fun(i), do: [i, i * i]

  def flat_map(enum) do
    Enum.flat_map(enum, &map_fun/1)
  end

  def map_flatten(enum) do
    enum |> Enum.map(&map_fun/1) |> List.flatten()
  end
end

The inputs in the benchmark module must now be a list of arguments.

defmodule Benchmark.Five do
  use BencheeDsl.Benchmark

  config(warmup: 1, time: 3, pre_check: true, print: [configuration: false])

  inputs(%{
    "Small" => [Enum.to_list(1..1_000)],
    "Big" => [Enum.to_list(1..100_000)]
  })

  job(&Benchmark.Jobs.flat_map/1)
  job(&Benchmark.Jobs.map_flatten/1, as: "map and flat")
end

Benchmark.Five.run()

Capture all jobs

The macro jobs/1 generates from each public function of a given module a job.

defmodule Benchmark.Six do
  use BencheeDsl.Benchmark

  config(warmup: 1, time: 3)

  inputs(%{
    "Small" => [Enum.to_list(1..1_000)],
    "Big" => [Enum.to_list(1..100_000)]
  })

  jobs(Benchmark.Jobs)
end

Benchmark.Six.run()

Benchee smart cell

BencheeDsl also brings the Benchee smart cell.

{:module, name, _binary, _bindings} =
  defmodule Benchmark do
    use BencheeDsl.Benchmark
    config(warmup: 1, time: 1, pre_check: true)
    inputs(%{"Small" => Enum.to_list(1..1000), "Big" => Enum.to_list(1..100_000)})

    defp map_fun(i) do
      [i, i * i]
    end

    job(flat_map(input)) do
      Enum.flat_map(input, &map_fun/1)
    end

    job(map_flatten(input)) do
      input |> Enum.map(&map_fun/1) |> List.flatten()
    end
  end

BencheeDsl.Livebook.benchee_config() |> name.run() |> BencheeDsl.Livebook.render()

Benchee style

If you have read everything up to here and still don't want to have a DSL, then Benchee alone will do.

defmodule MyMap do
  def flat_map(input) do
    Enum.flat_map(input, &map_fun/1)
  end

  def map_flatten(input) do
    input |> Enum.map(&map_fun/1) |> List.flatten()
  end

  defp map_fun(i), do: [i, i * i]
end

Benchee.run(
  %{
    "flat_map" => &MyMap.flat_map/1,
    "map.flatten" => &MyMap.map_flatten/1
  },
  warmup: 1,
  time: 3,
  inputs: %{
    "Small" => Enum.to_list(1..1_000),
    "Big" => Enum.to_list(1..100_000)
  }
)

This example shows also that we get the "same" results as in the other examples.