Skip to content

Latest commit

 

History

History
147 lines (113 loc) · 4.44 KB

benchmarks.md

File metadata and controls

147 lines (113 loc) · 4.44 KB

Benchmarks

Benchmarks are defined for raw C structs, Googles flatc generated C++ and the flatcc compilers C ouput.

These can be run with:

scripts/benchmark.sh

and requires a C++ compiler installed - the benchmark for flatcc alone can be run with:

test/benchmark/benchflatcc/run.sh

this only requires a system C compiler (cc) to be installed (and flatcc's build environment).

A summary for OS-X 2.2 GHz Haswell core-i7 is found below. Generated files for OS-X and Ubuntu are found in the benchmark folder.

The benchmarks use the same schema and dataset as Google FPL's FlatBuffers benchmark.

In summary, 1 million iterations runs at about 500-540MB/s at 620-700 ns/op encoding buffers and 29-34ns/op traversing buffers. flatc and flatcc are close enough in performance for it not to matter much. flatcc is a bit faster encoding but it is likely due to less memory allocation. Throughput and time per operatin are of course very specific to this test case.

Generated JSON parser/printer shown below, for flatcc only but for OS-X and Linux.

operation: flatbench for raw C structs encode (optimized)

elapsed time: 0.055 (s)
iterations: 1000000
size: 312 (bytes)
bandwidth: 5665.517 (MB/s)
throughput in ops per sec: 18158707.100
throughput in 1M ops per sec: 18.159
time per op: 55.070 (ns)

operation: flatbench for raw C structs decode/traverse (optimized)

elapsed time: 0.012 (s)
iterations: 1000000
size: 312 (bytes)
bandwidth: 25978.351 (MB/s)
throughput in ops per sec: 83263946.711
throughput in 1M ops per sec: 83.264
time per op: 12.010 (ns)

operation: flatc for C++ encode (optimized)

elapsed time: 0.702 (s)
iterations: 1000000
size: 344 (bytes)
bandwidth: 490.304 (MB/s)
throughput in ops per sec: 1425301.380
throughput in 1M ops per sec: 1.425
time per op: 701.606 (ns)

operation: flatc for C++ decode/traverse (optimized)

elapsed time: 0.029 (s)
iterations: 1000000
size: 344 (bytes)
bandwidth: 11917.134 (MB/s)
throughput in ops per sec: 34642832.398
throughput in 1M ops per sec: 34.643
time per op: 28.866 (ns)

operation: flatcc for C encode (optimized)

elapsed time: 0.626 (s)
iterations: 1000000
size: 336 (bytes)
bandwidth: 536.678 (MB/s)
throughput in ops per sec: 1597255.277
throughput in 1M ops per sec: 1.597
time per op: 626.074 (ns)

operation: flatcc for C decode/traverse (optimized)

elapsed time: 0.029 (s)
iterations: 1000000
size: 336 (bytes)
bandwidth: 11726.930 (MB/s)
throughput in ops per sec: 34901577.551
throughput in 1M ops per sec: 34.902
time per op: 28.652 (ns)

JSON benchmark

Note: this benchmark is only available for flatcc. It uses the exact same data set as above.

The benchmark uses Grisu3 floating point parsing and printing algorithm with exact fallback to strtod/sprintf when the algorithm fails to be exact. Better performance can be gained by enabling inexact Grisu3 and SSE 4.2 in build options, but likely not worthwhile in praxis.

operation: flatcc json parser and printer for C encode (optimized)

(encode means printing from existing binary buffer to JSON)

elapsed time: 1.407 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 513.068 (MB/s)
throughput in ops per sec: 710619.931
throughput in 1M ops per sec: 0.711
time per op: 1.407 (us)

operation: flatcc json parser and printer for C decode/traverse (optimized)

(decode/traverse means parsing json to flatbuffer binary and calculating checksum)

elapsed time: 2.218 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 325.448 (MB/s)
throughput in ops per sec: 450758.672
throughput in 1M ops per sec: 0.451
time per op: 2.218 (us)

JSON parsing and printing on same hardware in Virtual Box Ubuntu

Numbers for Linux included because parsing is significantly faster.

operation: flatcc json parser and printer for C encode (optimized)

elapsed time: 1.210 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 596.609 (MB/s)
throughput in ops per sec: 826328.137
throughput in 1M ops per sec: 0.826
time per op: 1.210 (us)

operation: flatcc json parser and printer for C decode/traverse

elapsed time: 1.772 (s)
iterations: 1000000
size: 722 (bytes)
bandwidth: 407.372 (MB/s)
throughput in ops per sec: 564227.736
throughput in 1M ops per sec: 0.564
time per op: 1.772 (us)