Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

README improvements #648

Merged
merged 3 commits into from
Jul 26, 2018
+60 −50
Merged

README improvements #648

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7ab1d66
Clarifications and cleaning of the core documentation.
dmah42 Jul 26, 2018
c92b782
Header cleanup
dmah42 Jul 26, 2018
701fe25
minor tweaks
dmah42 Jul 26, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
108 changes: 58 additions & 50 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,9 @@

A library to support the benchmarking of functions, similar to unit-tests.

Discussion group: https://groups.google.com/d/forum/benchmark-discuss
[Discussion group](https://groups.google.com/d/forum/benchmark-discuss)

IRC channel: https://freenode.net #googlebenchmark

[Known issues and common problems](#known-issues)
IRC channel: [freenode](https://freenode.net) #googlebenchmark

[Additional Tooling Documentation](docs/tools.md)

Expand Down Expand Up @@ -47,34 +45,31 @@ to `CMAKE_ARGS`.

For Ubuntu and Debian Based System

First make sure you have git and cmake installed (If not please install it)
First make sure you have git and cmake installed (If not please install them)

```
sudo apt-get install git
sudo apt-get install cmake
sudo apt-get install git cmake
```

Now, let's clone the repository and build it

```
git clone https://github.com/google/benchmark.git
cd benchmark
git clone https://github.com/google/googletest.git
# If you want to build tests and don't use BENCHMARK_DOWNLOAD_DEPENDENCIES, then
# git clone https://github.com/google/googletest.git
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=RELEASE
make
```

We need to install the library globally now
If you need to install the library globally

```
sudo make install
```

Now you have google/benchmark installed in your machine
Note: Don't forget to link to pthread library while building

## Stable and Experimental Library Versions

The main branch contains the latest stable version of the benchmarking library;
Expand All @@ -87,15 +82,16 @@ to use, test, and provide feedback on the new features are encouraged to try
this branch. However, this branch provides no stability guarantees and reserves
the right to change and break the API at any time.

## Prerequisite knowledge

Before attempting to understand this framework one should ideally have some familiarity with the structure and format of the Google Test framework, upon which it is based. Documentation for Google Test, including a "Getting Started" (primer) guide, is available here:
https://github.com/google/googletest/blob/master/googletest/docs/primer.md
## Further knowledge

It may help to read the [Google Test documentation](https://github.com/google/googletest/blob/master/googletest/docs/primer.md)
as some of the structural aspects of the APIs are similar.

## Example usage
### Basic usage
Define a function that executes the code to be measured.
Define a function that executes the code to be measured, register it as a
benchmark function using the `BENCHMARK` macro, and ensure an appropriate `main`
function is available:

```c++
#include <benchmark/benchmark.h>
Expand Down Expand Up @@ -123,7 +119,21 @@ Don't forget to inform your linker to add benchmark library e.g. through
`BENCHMARK_MAIN();` at the end of the source file and link against
`-lbenchmark_main` to get the same default behavior.

The benchmark library will reporting the timing for the code within the `for(...)` loop.
The benchmark library will reporting the timing for the code within the
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

library will reporting the timing

Something is missing between will and reporting.

`for(...)` loop.

#### Platform-specific libraries
When the library is built using GCC it is necessary to link with the pthread
library due to how GCC implements `std::thread`. Failing to link to pthread will
lead to runtime exceptions, not linker errors. See [issue #67](https://github.com/google/benchmark/issues/67)
for more details. You can link to pthread by adding `-lpthread` to your linker
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think -pthread is preferable. It's like -lpthread but the compiler puts it in the right place. So you can even put the flag before your object files, and they'll still link.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will lead to runtime exceptions, not linker errors. See [issue #67]

Hey now! Libc++ has the decency to fail to link!

command.

If you're running benchmarks on Windows, the shlwapi library (`-lshlwapi`) is
also required.

If you're running benchmarks on solaris, you'll want the kstat library linked in
too (`-lkstat`).

### Passing arguments
Sometimes a family of benchmarks can be implemented with just one routine that
Expand Down Expand Up @@ -522,15 +532,7 @@ order to manually set the time unit, you can specify it manually:
BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);
```

## Controlling number of iterations
In all cases, the number of iterations for which the benchmark is run is
governed by the amount of time the benchmark takes. Concretely, the number of
iterations is at least one, not more than 1e9, until CPU time is greater than
the minimum time, or the wallclock time is 5x minimum time. The minimum time is
set as a flag `--benchmark_min_time` or per benchmark by calling `MinTime` on
the registered benchmark object.

## Reporting the mean, median and standard deviation by repeated benchmarks
### Reporting the mean, median and standard deviation by repeated benchmarks
By default each benchmark is run once and that single result is reported.
However benchmarks are often noisy and a single result may not be representative
of the overall behavior. For this reason it's possible to repeatedly rerun the
Expand Down Expand Up @@ -810,8 +812,29 @@ BM_memcpy/32 12 ns 12 ns 54687500
BM_memcpy/32k 1834 ns 1837 ns 357143
```

## Runtime and reporting considerations
When the benchmark binary is executed, each benchmark function is run serially.
The number of iterations to run is determined dynamically by running the
benchmark a few times and measuring the time taken and ensuring that the
ultimate result will be statistically stable. As such, faster benchmark
functions will be run for more iterations than slower benchmark functions, and
the number of iterations is thus reported.

## Output Formats
In all cases, the number of iterations for which the benchmark is run is
governed by the amount of time the benchmark takes. Concretely, the number of
iterations is at least one, not more than 1e9, until CPU time is greater than
the minimum time, or the wallclock time is 5x minimum time. The minimum time is
set per benchmark by calling `MinTime` on the registered benchmark object.

Average timings are then reported over the iterations run. If multiple
repetitions are requested using the `--benchmark_repetitions` command-line
option, or at registration time, the benchmark function will be run several
times and statistical results across these repetitions will also be reported.

As well as the per-benchmark entries, a preamble in the report will include
information about the machine on which the benchmarks are run.

### Output Formats
The library supports multiple output formats. Use the
`--benchmark_format=<console|json|csv>` flag to set the format type. `console`
is the default format.
Expand Down Expand Up @@ -879,14 +902,15 @@ name,iterations,real_time,cpu_time,bytes_per_second,items_per_second,label
"BM_SetInsert/1024/10",106365,17238.4,8421.53,4.74973e+06,1.18743e+06,
```

## Output Files
### Output Files
The library supports writing the output of the benchmark to a file specified
by `--benchmark_out=<filename>`. The format of the output can be specified
using `--benchmark_out_format={json|console|csv}`. Specifying
`--benchmark_out` does not suppress the console output.

## Debug vs Release
By default, benchmark builds as a debug library. You will see a warning in the output when this is the case. To build it as a release library instead, use:
By default, benchmark builds as a debug library. You will see a warning in the
output when this is the case. To build it as a release library instead, use:

```
cmake -DCMAKE_BUILD_TYPE=Release
Expand All @@ -898,16 +922,11 @@ To enable link-time optimisation, use
cmake -DCMAKE_BUILD_TYPE=Release -DBENCHMARK_ENABLE_LTO=true
```

If you are using gcc, you might need to set `GCC_AR` and `GCC_RANLIB` cmake cache variables, if autodetection fails.
If you are using clang, you may need to set `LLVMAR_EXECUTABLE`, `LLVMNM_EXECUTABLE` and `LLVMRANLIB_EXECUTABLE` cmake cache variables.

## Linking against the library
If you are using gcc, you might need to set `GCC_AR` and `GCC_RANLIB` cmake
cache variables, if autodetection fails.

When the library is built using GCC it is necessary to link with `-pthread`,
due to how GCC implements `std::thread`.

For GCC 4.x failing to link to pthreads will lead to runtime exceptions, not linker errors.
See [issue #67](https://github.com/google/benchmark/issues/67) for more details.
If you are using clang, you may need to set `LLVMAR_EXECUTABLE`,
`LLVMNM_EXECUTABLE` and `LLVMRANLIB_EXECUTABLE` cmake cache variables.

## Compiler Support

Expand Down Expand Up @@ -937,14 +956,3 @@ sudo cpupower frequency-set --governor performance
./mybench
sudo cpupower frequency-set --governor powersave
```

# Known Issues

### Windows with CMake

* Users must manually link `shlwapi.lib`. Failure to do so may result
in unresolved symbols.

### Solaris

* Users must explicitly link with kstat library (-lkstat compilation flag).