Hashtable latencies

This benchmark measures the latency of a web service that stores a large (250K item) hashtable, in multiple languages.

Garbage collected languages are of greater interest, and the Swift version is mostly used as a reference of what would be ideal.

Rules for implementations.

• Each HTTP request will add one new item (and remove one if count > 250K)
• Each item will be a new 1KB buffer with every byte initialized to a value.
• Programs will listen on port 8080 and respond with status 200, body "OK"

Structure

• All program directories are listed in programs.txt
• Every directory should contain build.sh and run.sh
• wrk2 must be installed and linked in $PATH as wrk2

Tests

• Warmup: 10K req/s are sent for 60s (initial hash table content)
• Test 033: 10K req/s are sent by 33 clients concurrently for 60s
• Test 333: 10K req/s are sent by 333 clients concurrently for 60s

Output:

• All reports are in reports directory.
• Charts from both tests can be generated by using hdrhistogram

Current results

33 clients test:

333 clients test:

Program notes

• reason - This is OCaml with a new syntax. Follow the install instructions
• node - Requires node 6.x (Buffer.alloc API)
• haskell - Should build easily if you have stack
• swift-zewo - You should setup swiftenv to the right snapshot

Conclusions

Want to build low-latency, memory-intensive services in a GCed language? Use OCaml (or Reason if you prefer the syntax).

OCaml GC notes

Other things that were attempted to make the latency worse for OCaml:

• random buffer sizes (did not have any noticable increase, <5%)
• longer running time (5 minutes): another 10% latency increase.
• combining above: total latency increase of 15%

Things to try:

• delete keys out of order (might increase heap compaction duration?)