This document describes the artifact for our SOSP 2021 paper on HeMem. HeMem is a tiered main memory management system designed from scratch for commercially available NVM and the big data applications that use it. HeMem manages tiered memory asynchronously, batching and amortizing memory access tracking, migration, and associated TLB synchronization overheads. HeMem monitors application memory use by sampling memory access via CPU events, rather than page tables. This allows HeMem to scale to terabytes of memory, keeping small and ephemeral data structures in fast memory, and allocating scarce, asymmetric NVM bandwidth according to access patterns. Finally, HeMem is flexible by placing per-application memory management policy at user-level.
apps/contains the application benchmarks evaluated with HeMemmicrobenchmarks/contains the GUPS microbenchmark used to evaluate HeMemsrc/contains the source code of HeMemsrc/policiescontains extra memory policies used for testing HeMem, such as a page-table based LRU policy
Hoard/contains the Hoard memory allocator that HeMem depends onlinux/contains the linux kernel version required to run HeMem
You may set up HeMem to run on your own machine provided you have Intel Optane NVM. HeMem uses /dev/dax files to represent DRAM and NVM. Some additional setup is required for setting up the DRAM and NVM /dev/dax files to run HeMem.
To set up the /dev/dax file representing DRAM, follow the instructions here in order to reserve a block of DRAM at machine startup to represent the DRAM /dev/dax file. HeMem reserves its 140GB of DRAM in this way (enough for its 128GB of reserved DRAM plus some metadata needed for ndctl). If your machine has multiple NUMA nodes, ensure that the block of DRAM you reserve is located on the same NUMA node that has NVM. Do not follow the last set of instructions from pmem.io on setting up a file system on the reserved DRAM. Instead, set up a /dev/dax file to represent it:
- First, determine the name of the namespace representing the reserved DRAM:
ndctl list --human
- You should see your reserved DRAM. If multiple namespaces are listed, some represent NVM namespaces (described below). You should be able to differentiate the DRAM namespace based on size. Your DRAM namespace is likely in
fsdaxmode. Change the namespace over todevdaxmode using the following command (in this example, the DRAM namespace is callednamespace0.0):
sudo ndctl create-namespace -f -e namespace0.0 --mode=devdax --align 2M
- Make note of the
chardevname of the DRAM/dev/daxfile. This will be used to tell HeMem which/dev/daxfile represents DRAM. If this is different fromdax0.0, then you will need to edit thesrc/hemem.hfileDRAMPATHmacro to point it towards your actual DRAM/dev/daxfile.
To set up the /dev/dax file representing NVM, ensure that your machine has NVM in App Direct mode. If you do not already have namespaces representing NVM, then you will need to create them. Follow these steps:
- List the regions available on your machine:
ndctl list --regions --human
- Note which regions represent NVM. You can differentiate them from the reserved DRAM region based on size or via the
persistence_domainfield, which, for NVM, will readmemory_controller. Pick the region that is on the same NUMA node as your reserved DRAM. In this example, this is "region1". Create a namespace over this region:
ndctl create-namespace --region=1 --mode=devdax
- Make note of the
chardevname of the NVM/dev/daxfile. This will be used to tell HeMem which/dev/daxfile represents NVM. If this is different fromdax1.0, then you will need to edit thesrc/hemem.hfileNVMPATHmacro to point it towards your actual NVM/dev/daxfile.
To build HeMem, you must first build the linux kernel HeMem depends on. Build, install, and run the kernel located in the linux/ directory.
Next, HeMem depends on Hoard. Follow the instructions to build the Hoard library located in the Hoard/ directory.
HeMem also depends on libsyscall_intercept to intercept memory allocation system calls. Follow the instructions to build and install libsyscall_intercept here.
Once the proper kernel version is running, the /dev/dax files have been set up, and all dependencies have been installed, HeMem can be built with the supplied Makefile by typing make from the src/ directory.
You will likely need to add the paths to the build HeMem library and the Hoard library to your LD_LIBRARY_PATH variable:
export LD_LIBRARY_PATH=path/to/hemem/lib:/path/to/Hoard/lib:$LD_LIBRARY_PATH
You may also need to increase the number of allowed mmap ranges:
echo 1000000 > /proc/sys/vm/max_map_count
HeMem requires the user be root in order to run. Applications can either be linked with Hemem or run unmodified via the LD_PRELOAD environment variable:
LD_PRELOAD=/path/to/hemem/lib.so ./foo [args]
A Makefile is provided to build the GUPS microbenchmarks.
To reproduce the Uniform GUPS results, run the run-random.sh script. Results will be printed to the random.txt file. The throughput results shown in the paper are the "GUPS" lines.
To reproduce the Hotset GUPS results, run the run.sh script. Results will be printed to the results.txt file. The throughput results shown in the paper are the "GUPS" lines.
To reproduce the Instantaneous GUPS results, run the run-instantaneous.sh script. Results will be printed to the tot_gups.txt file.
Applications tested with HeMem are located in the apps/ directory.
The Silo application can be found in the apps/silo_hemem/silo directory.. Run the provided run_batch.sh script. Results will be in the batch/results.txt file. The reported throughput numbers are numbers in the first column of the file.
The FlexKVS application can be found in the apps/flexkvs directory. These results require a separate machine for the clients.
The GapBS application can be found in the apps/gapbs directory. To run the BC algorithm reported in the paper, you may run the following command:
LD_PRELOAD=/path/to/hemem/lib ./bc -g <scale>
which will run the bc algorithm with HeMem on a graph with 2^scale vertices.