Developer-Guide.md

OAP Developer Guide

NOTE: This document is for the whole project of OAP about building, you can learn more detailed information from every module's document.

• SQL Index and Data Source Cache
• RDD Cache PMem Extension
• Shuffle Remote PMem Extension
• Remote Shuffle

OAP Building

To clone OAP project, use

git clone -b <tag-version> https://github.com/Intel-bigdata/OAP.git
cd OAP

Prerequisites for Building

OAP is built using Apache Maven. You need to install the required packages on the build system listed below. To enable Shuffle Remote PMem extension, you must configure and validate RDMA in advance, you can refer to Shuffle Remote PMem Extension Guide for more details.

• Cmake
• Memkind
• Vmemcache
• HPNL
• PMDK
• GCC > 7

You can use the following command under the folder dev to automatically install these dependencies.

source $OAP_HOME/dev/prepare_oap_env.sh
prepare_maven

You can also use command like prepare_cmake to install the specified dependencies after executing the command "source prepare_oap_env.sh". Use the following command to learn more.

oap_build_help

If you want to automatically install all dependencies,use:

prepare_all

Building

If you use "prepare_oap_env.sh" to install GCC, or your GCC is not installed in the default path, please export CC (and CXX) before calling maven.

export CXX=$OAPHOME/dev/thirdparty/gcc7/bin/g++
export CC=$OAPHOME/dev/thirdparty/gcc7/bin/gcc

To build OAP package, use

mvn clean -DskipTests package

Building Specified Module

mvn clean -pl com.intel.oap:oap-cache -am package

Running Test

To run all the tests, use

mvn clean test

Running Specified Test

mvn clean -pl com.intel.oap:oap-cache -am test

OAP Building with PMem

Prerequisites for building with PMem support

If you want to use OAP-CACHE with PMem, you must finish steps of "Prerequisites for building" to ensure all dependencies have been installed .

Building package

You need to add -Ppersistent-memory to build with PMem support. For noevict cache strategy, you also need to build with -Ppersistent-memory parameter.

mvn clean -q -Ppersistent-memory -DskipTests package

for vmemcache cache strategy, please build with command:

mvn clean -q -Pvmemcache -DskipTests package

You can build with command to use all of them:

mvn clean -q -Ppersistent-memory -Pvmemcache -DskipTests package

OAP Packaging

If you want to generate a release package after you mvn package all modules, use the following command under the directory dev, then you can find a tarball named oap-product-$VERSION-bin-spark-2.4.4.tar.gz in dev/release-package dictionary.

sh make-distribution.sh