TigerBeetle is a purpose-built financial accounting database, designed for high-throughput low-latency two-phase prepare/commit transfers between accounts.
Watch a video introduction of TigerBeetle given to the Interledger community on 25 November 2020:
Processing events that impact the balance of an account must be done serially, in the correct order and reliably. For this reason, despite the trend to scale modern applications by distributing work across parallel threads/processes/machines, it remains difficult to scale a ledger or accounting system without sacrificing performance or safety.
For example, processing a balance update event involves:
- validating the event,
- processing any business rules that must consider the current balance before processing the event and the new balance after the event,
- updating the balance
- persisting the updated balance, and
- notifying subscribers of the updated balance and that the event has been processed.
While investigating a few existing systems it became clear that the majority of architectures cobble together generic databases (relational or NoSQL, on-disk or in-memory) with business logic enforced in the application code. This separation between data and code, persistence and logic, invites the worst of distributed system problems: network delays, multiple roundtrips for a single balance update, clock skew, or cache incoherency etc.
Furthermore, these systems may achieve performance but at the expense of reliability, especially in the face of hardware failure, corruption and misdirected writes, a compromise that is unacceptable for a system-of-record for financial accounts.
Our survey led us to conclude that, while there are mechanisms available to shard account balances to parallelize updates and improve performance, there are still significant performance gains to be had in designing a database that is purpose built for storing balances and processing updates in a reliable manner.
In the month of July 2020, we developed a prototype of TigerBeetle in Node as a performance sketch to measure the basic components of the design (batching, TCP protocol, cryptographic checksums everywhere, fsync journalling, in-memory business logic and hash table operations). ProtoBeetle ran at 200,000 transfers per second on our office laptops, supporting our back-of-the-envelope numbers.
We then integrated ProtoBeetle into Mojaloop and our reference minimum deployment cluster of Mojaloop went from 76 TPS on MySQL to 1757 TPS on ProtoBeetle. A single stateless Mojaloop pod was unable to saturate ProtoBeetle. Most of the throughput was spent converting Mojaloop's individual HTTP requests into TCP batches.
Watch a 10-minute talk introducing ProtoBeetle.
After ProtoBeetle, from September through October 2020, we knuckled down and rewrote TigerBeetle in C/Zig to create the alpha version of TigerBeetle, using io_uring as a foundation for fast I/O.
TigerBeetle's Zig implementation of io_uring was submitted for addition to the Zig standard library.
Watch a presentation of TigerBeetle given to the Interledger community on 25 November 2020.
The beta version of TigerBeetle is now under active development and our design document details our design decisions regarding performance and safety, and where we want to go regarding accounting features.
Prerequisites: The current beta version of TigerBeetle targets Linux and takes advantage of the latest asynchronous IO capabilities of the Linux kernel v5.6 and newer, via io_uring. As such it can only be used on recent versions of Linux with an updated kernel.
Later portable versions of TigerBeetle may supplement io_uring with kqueue for macOS and FreeBSD support, or IOCP for Windows support.
git clone https://github.com/coilhq/tigerbeetle.git
cd tigerbeetle
scripts/install.shIf you want to run Parallels on macOS on an M1 chip, we recommend the server install image of Ubuntu 20.10 Groovy Gorilla for ARM64, which ships with Linux 5.8 and which will support installing Parallels Tools easily.
With TigerBeetle installed, you are ready to benchmark!
scripts/benchmark.shIf you encounter any benchmark errors, please send us the resulting benchmark.log.
Launch a TigerBeetle cluster on your local machine by running each of these commands in a new terminal tab:
./tigerbeetle --cluster-id=0a5ca1ab1ebee11e --replica-addresses=3001,3002,3003 --replica-index=0
./tigerbeetle --cluster-id=0a5ca1ab1ebee11e --replica-addresses=3001,3002,3003 --replica-index=1
./tigerbeetle --cluster-id=0a5ca1ab1ebee11e --replica-addresses=3001,3002,3003 --replica-index=2
Run the TigerBeetle binary to see all command line arguments:
./tigerbeetle --help-
tigerbeetle-node is a TigerBeetle Node.js client written in TypeScript (and Zig with Node's N-API for ABI stability).
-
client.zig is a TigerBeetle Zig client.
-
demo.zig is a lightweight TigerBeetle client for demonstration purposes only, which we used to create six demos you can work your way through and modify to explore TigerBeetle's commands.
Along the way, we also put together a series of performance demos and sketches to get you comfortable building TigerBeetle, show how low-level code can sometimes be easier than high-level code, help you understand some of the key components within TigerBeetle, and enable back-of-the-envelope calculations to motivate design decisions.
You may be interested in:
- demos/protobeetle, how batching changes everything.
- demos/bitcast, how Zig makes zero-overhead network deserialization easy, fast and safe.
- demos/io_uring, how ring buffers can eliminate kernel syscalls, reduce server hardware requirements by a factor of two, and change the way we think about event loops.
- demos/hash_table, how linear probing compares with cuckoo probing, and what we look for in a hash table that needs to scale to millions (and billions) of account transfers.
Copyright 2020-2021 Coil Technologies, Inc
Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.