C++ templated collections library for the LMDB database
Trivial ultra low-latency zero-copy storage for anywhere from small to very large volumes of data.
Header only library, for easy inclusion in projects: just put ./include
on your
compiler include path.
Requires C++11 or later. Tested under 64 bit Linux and Windows msys2; should work fine with MSVC.
We typically map a POD class to another POD class, or a POD class to an array of POD classes; both are quite similar.
Here we demonstrate the latter:
#include <lmdbcols.hpp>
// DB file on disk is "my_data.db"; created if doesn't yet exist
// (Sets max DB file size to 1GB; see ctr definition if you want another value)
auto env = lmdbcols::DatabaseEnvironment{ "my_data.db" };
// Key and Value types to store in the DB
using PathID = uint64_t;
struct GeoPos {double lat, lon;};
// Store our data in a named LMDB database (in the file) called "paths"
auto pathsDB = lmdbcols::MapDB_Pod_PodArray<PathID, GeoPos>{ "paths" };
// Write path
{
// All interaction with the db happens through transactions
auto txn = env.openWriteTxn();
// You can put() from any contiguous array in memory
auto onePath = std::vector<GeoPos>{ {0,0}, {1,0}, {1,1}, {0,0} };
pathsDB.put( txn, 123, &onePath[0], onePath.size() );
// RAII calls abort() on txn if you don't commit before scope end
txn.commit();
}
// Read path
{
// Use read transactions whenever possible, for perf
auto txn = env.openReadTxn();
// Note that the data returned here is memory mapped, so only materialised
// from disk to RAM when you dereference a pointer inside it
auto path = pathsDB.get( txn, 123 );
// This obviously materialises everything
std::for_each( std::begin(path), std::end(path),
[](auto &gp){
printf("%f, %f\n", gp.lat, gp.lon);} );
}
You can find a lot more detais as to what's going on in the comments in lmdbcols.hpp.
By default LMDB offers very loose alignment guarantees on where keys and values are stored in memory.
In practice, since keys and values are stored contiguously, you can impose stricter guarandees by making sure that every key and every value in a single DB has as size being a multiple of the required value.
We take 8 bytes as a goldilocks position, so to use the standard MapDB_xxx classes the types you template them with must have sizeof a multiple of 8.
If this isn't easy to do, see the EightPadded type, which will pad another type out to the nearest 8 byte boundary with zeros.
Or just use the AutoPadded variants of the MapDB classes, which transparently use the EightPadded wrapper for you. See the (extensive) comments in lmdbcols.hpp if you want more information.
Note that e.g. if you have a long double
in a struct that's your value type,
you'll need to make sure yourself that the key type is also a multiple of 16
in size.
All project files are copyright (c) 2017 Inkblot Software Limited.
This project is released under the Mozilla Public License v2.0.
This means you can include the library in your projects (static or dynamic linking) without having to change your license or reveal your code, just like MIT.
But if you want to modify this library itself, all the derived versions of it stay under MPLv2.
LMDB (or github) is an embedded multi-database key-value store, written in C, which provides the best performance for most read-biased workloads, by some margin. The API is very clean, but not ideal for use
(lmdbxx)[https://github.com/bendiken/lmdbxx/] provides a superb C++ wrapper around LMDB, taking advantage of both RAII and exceptions. It doesn't impose any specific structure on how you should store data in the DB.
You need both to build with lmdbcols:
-
LMDB is easiest found in distro package managers. Otherwise it's just a few source files, so you can easily download the source and include the files in your build process: https://github.com/LMDB/lmdb
-
lmdbxx is a header only C++ library. For convenience we bundle it with lmdbcols, so just add
libs/lmdbxx
to your compiler include path
The provided CMakeLists.txt file will build a program called run_tests.
Build and run this with a file path to a not-yet-existing scratch db for its first argument. Test failures register as assert() failures.