Commit all source code and scripts

.gitignore

@@ -0,0 +1,58 @@
+# Prerequisites
+*.d
+
+# Compiled Object files
+*.slo
+*.lo
+*.o
+*.obj
+
+# Precompiled Headers
+*.gch
+*.pch
+
+# Compiled Dynamic libraries
+*.so
+*.dylib
+*.dll
+
+# Fortran module files
+*.mod
+*.smod
+
+# Compiled Static libraries
+*.lai
+*.la
+*.a
+*.lib
+
+# Executables
+*.exe
+*.out
+*.app
+
+# CMake Ignore files
+CMakeLists.txt.user
+CMakeCache.txt
+CMakeFiles
+CMakeScripts
+Testing
+Makefile
+cmake_install.cmake
+install_manifest.txt
+compile_commands.json
+CTestTestfile.cmake
+_deps
+
+# Build Files
+build/
+
+# IDE Files
+.vscode/
+tags
+testcase*
+**/__pycache__/**
+*.pdf
+
+breakdown*
+*.png

CMakeLists.txt

@@ -0,0 +1,60 @@
+cmake_minimum_required(VERSION 3.5.1)
+
+project(serverless-dedup-services C CXX)
+
+# Set DEBUG mode
+set(CMAKE_BUILD_TYPE Debug)
+
+# Include the common cmake file to find grpc and protobuf packages
+include(cmake/common.cmake)
+include(cmake/GenerateCPP.cmake)
+
+# Include Boost libraries
+SET(Boost_USE_STATIC_LIBS ON)
+FIND_PACKAGE(Boost 1.75 COMPONENTS log REQUIRED)
+
+
+set(PROTOS
+  ${CMAKE_CURRENT_SOURCE_DIR}/protos/main.proto
+  ${CMAKE_CURRENT_SOURCE_DIR}/protos/client.proto
+  ${CMAKE_CURRENT_SOURCE_DIR}/protos/controller.proto
+)
+
+set(PROTO_SRC_DIR ${CMAKE_CURRENT_BINARY_DIR}/proto-src)
+file(MAKE_DIRECTORY ${PROTO_SRC_DIR})
+
+# Run gRPC and Protobuyf compilers to generate C++ files
+generate_cpp(
+  GRPC_SRCS
+  GRPC_HDRS
+  PROTO_SRCS
+  PROTO_HDRS
+  ${PROTO_SRC_DIR}
+  ${PROTOS})
+
+# Include the genrated *.pb.h, *.grpc.pb.h files
+include_directories(${PROTO_SRC_DIR})
+
+# dedup_grpc_proto - A common proto library that can be used by all executables
+add_library(dedup_grpc_proto
+  ${GRPC_SRCS}
+  ${GRPC_HDRS}
+  ${PROTO_SRCS}
+  ${PROTO_HDRS})
+target_link_libraries(dedup_grpc_proto
+  ${_REFLECTION}
+  ${_GRPC_GRPCPP}
+  ${_PROTOBUF_LIBPROTOBUF})
+
+set(COMMON_HEADERS_DIR ${PROJECT_SOURCE_DIR}/include)
+include_directories(${COMMON_HEADERS_DIR})
+
+# Move the config files into the cmake directory as well
+file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/config/ DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/config/)
+
+# Move the shell scripts into the cmake directory as well
+file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/scripts/ DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/scripts/)
+
+add_subdirectory(dedup-controller)
+add_subdirectory(dedup-service)
+add_subdirectory(rdma)

README.md

@@ -0,0 +1,125 @@
+# Medes
+
+Medes is a serverless framework that employs memory deduplication to reduce memory footprints of a warm container.  
+It leverages the fact that the warm sandboxes running on serverless platforms have a high fraction of duplication in their memory footprints. It exploits these redundant chunks to develop a new sandbox state, called a dedup state, that is more memory-efficient than the warm state and faster to restore from than the cold state.  
+Hence, Medes breaks the rigid trade-off and allows operators to navigate the memory-performance trade-off space smoothly. 
+
+## Implementation Overview
+
+To employ memory deduplication, Medes uses C/R mechanism to fetch the raw memory dump of a warm container.
+Then it chooses a representative set of 64B chunks from each page of the dump, and looks up each one of them in a global hash table, that is stored on a global controller.
+For each such page of the container to be dedup-ed, this controller employs a simple heuristic to provide addresses of pages (called `base page's) with respect to which, that page shall be deduplicated.
+Thereafter, the pages on the dedup-ed container are stored as a patch and a pointer to their respective base pages.
+This information is locally stored at the node and at restore, the base pages are read and original pages of the dedup container are restored.
+
+More details of the architecture, system design, optimizations and implementation are available in our paper [here](https://doi.org/10.1145/3492321.3524272).
+
+## Directory Structure
+
+```bash
+.
+├── CMakeLists.txt
+├── cmake
+├── config
+├── dedup-controller
+├── dedup-service
+├── evaluation
+├── include
+├── motivation
+├── protos
+├── rdma
+└── scripts
+```
+
+The library for the controller is implemented in `dedup-controller/` while the library for dedup agent on each individual machine is implemented in `dedup-service/`. The dedup agent makes use of RDMA to read local as well as remote pages, a wrapper for which, is available in `rdma/`.  
+General include headers are segregated in `include/` directory, while the proto files are located in a common location: `protos/`.  
+All scripts needed during the execution of the framework are placed in `scripts/`.  
+`motivation/` houses the scripts to run and plot motivation experiments, and all other plotting scripts are placed in `evaluation/`.
+
+## Build and Compile
+
+### Prerequisites
+
+The prerequisites can be automatically installed using the `cloudlab/config.sh` script. Medes requires the following:
+
+- CMake
+- gRPC
+- Boost (atleast v1.75.0)
+- Python and pip
+- Docker (v19.03.12)
+- xDelta3
+- Mellanox OFED (v5.4-3.0.3) and an RNIC
+- Openwhisk Python Runtime - modified for Medes. Repo [here](https://github.com/DivyanshuSaxena/openwhisk-runtime-python/tree/dedup)
+- CRIU - modified and optimized for Medes. Repo [here](https://github.com/DivyanshuSaxena/criu/tree/mod-criu)
+
+Run `./install.sh` to install all of these dependencies in one go. (**Note:** By default, it will install all dependencies in `$HOME`.)
+
+Finally, to compile, from the top directory, run:
+
+```console
+./build.sh
+```  
+
+Read `build.sh` to get more information. It assumes that cmake is installed in `$HOME/.local/` and xdelta is installed in `$HOME/xdelta-3.1.0/xdelta3`. If you used the installation script `install.sh`, then these paths are already correct.
+
+## Run
+
+### Configurations
+
+The `config/` folder has two important configuration files: `agent.ini`, used to configure the Dedup agents, and `controller.ini`, used to configure the global controller. Respective files have detailed comments about how to use them.
+
+#### Important Parameters
+
+The above mentioned .ini files expose a few important parameters, as discussed below:
+
+1. `policy`: Chooses which policy is run by the controller, for managing containers.
+2. `constraint`: If the Medes policy is chosen, then the platform admin has the flexibility to choose whether they want Medes to function under a memory constraint (get best latency under a fixed memory budget), or a latency constraint (occupy least amount of memory while satisfying a latency bound).
+3. `dedupperbase`: The threshold for the ratio of dedup containers to base containers. If the ratio exceeds this threshold, another base container is demarcated.
+4. `memcap`: The software-defined limit on the memory usage of each node (in MB).
+1. `chunksperpage`: This represents the cardinality of the fingerprint of a page. A higher value implies more chunks shall be sampled from the page, leading to a better duplication identification, but increased communication overheads. _(Default: 5)_
+2. `idletime`: This represents the period of time a container remains in the same state (whether warm or dedup), after which the dedup agent shall seek an updated decision for the container from the policy controller.
+3. `adaptive`: If set to true(=1), the adaptive keep-alive policy shall be used.
+
+Cluster configuration is required in the `config/cluster.json` file. The format is as follows:
+
+```
+{
+    "controller": {
+        "addr": "<ip-address-of-controller>",
+        "port": <port>
+    },
+    "memory_nodes": [
+        {
+            "machine_id": 0,
+            "addr": "<ip-address-of-machine-with-id-0>",
+            "port": <rdma-server-port>
+        },
+        ...
+    ],
+    "grpc_nodes": [
+        {
+            "machine_id": 0,
+            "addr": "<ip-address-of-machine-with-id-0>",
+            "port": <grpc-port>
+        },
+        ...
+    ]
+}
+```
+
+To run the controller, from the `cmake/build` directory:
+
+```console
+./dedup-controller/controller <number-of-threads> <path-to-requests-file>
+```
+
+To run the dedup agent, from the `cmake/build` directory:
+
+```console
+./dedup-service/dedup_appl <node-id> <number-of-threads>
+```
+
+## Contributions and Contacts
+
+This project is licensed under the [Apache License 2.0](LICENSE).  
+For any questions, feel free to contact [Divyanshu Saxena](https://divyanshusaxena.github.io) or [Tao Ji](mailto:taoji@cs.utexas.edu), file issues or submit PRs.

build.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+mkdir -p cmake/build
+pushd cmake/build
+cmake -DCMAKE_PREFIX_PATH=$HOME/.local/ -DCMAKE_BUILD_TYPE=Debug -DXDELTA_DIR=$HOME/xdelta-3.1.0/xdelta3 ../..
+make -j$(getconf _NPROCESSORS_ONLN)
+popd

cmake/GenerateCPP.cmake

@@ -0,0 +1,64 @@
+#
+# Generates C++ sources from the .proto files
+#
+# grpc_generate_cpp (<SRCS> <HDRS> <DEST> [<ARGN>...])
+#
+#  GSRCS, PSRCS - variable to define with autogenerated source files
+#  GHDRS, PSRCS - variable to define with autogenerated header files
+#  DEST - directory where the source files will be created
+#  ARGN - .proto files
+#
+function(GENERATE_CPP GSRCS GHDRS PSRCS PHDRS DEST)
+  if(NOT ARGN)
+    message(SEND_ERROR "Error: GENERATE_CPP() called without any proto files")
+    return()
+  endif()
+
+  set(PROTO_PATH)
+  # Create an include path for each file specified
+  foreach(FIL ${ARGN})
+    get_filename_component(ABS_FIL ${FIL} ABSOLUTE)
+    get_filename_component(ABS_PATH ${ABS_FIL} PATH)
+
+    list(FIND PROTO_PATH ${ABS_PATH} _contains_already)
+    if(${_contains_already} EQUAL -1)
+        list(APPEND PROTO_PATH ${ABS_PATH})
+    endif()
+  endforeach()
+
+  message(STATUS "Using Proto Path: ${PROTO_PATH}")
+
+  set(G_SRCS)
+  set(G_HDRS)
+  set(P_SRCS)
+  set(P_HDRS)
+
+  foreach(FIL ${ARGN})
+    get_filename_component(ABS_FIL ${FIL} ABSOLUTE)
+    get_filename_component(FIL_WE ${FIL} NAME_WE)
+
+    list(APPEND G_SRCS "${DEST}/${FIL_WE}.grpc.pb.cc")
+    list(APPEND G_HDRS "${DEST}/${FIL_WE}.grpc.pb.h")
+    list(APPEND P_SRCS "${DEST}/${FIL_WE}.pb.cc")
+    list(APPEND P_HDRS "${DEST}/${FIL_WE}.pb.h")
+
+    add_custom_command(
+      OUTPUT "${DEST}/${FIL_WE}.grpc.pb.cc"
+             "${DEST}/${FIL_WE}.grpc.pb.h"
+             "${DEST}/${FIL_WE}.pb.cc"
+             "${DEST}/${FIL_WE}.pb.h"
+      COMMAND ${_PROTOBUF_PROTOC}
+      ARGS --grpc_out "${DEST}"
+        --cpp_out "${DEST}"
+        -I "${PROTO_PATH}"
+        --plugin=protoc-gen-grpc="${_GRPC_CPP_PLUGIN_EXECUTABLE}"
+        "${ABS_FIL}"
+      DEPENDS "${ABS_FIL}"
+      COMMENT "Running C++ gRPC and Protobuf compiler on ${FIL}")
+  endforeach()
+
+  set(${GSRCS} ${G_SRCS} PARENT_SCOPE)
+  set(${GHDRS} ${G_HDRS} PARENT_SCOPE)
+  set(${PSRCS} ${P_SRCS} PARENT_SCOPE)
+  set(${PHDRS} ${P_HDRS} PARENT_SCOPE)
+endfunction()