Python Wrapping

Introduction

We have started wrapping our open source c++ libraries with python wrappers. This exposes our optimized libraries to a python interface allowing us to use them in conjunction with Numpy and Conda.

Organization

The files for the wrapping are all located in a 'python' directory in the source directory. The files all end in _py or _pyt. This helps distinguish them from all the other source files. The _py is for the c++ files used to generate the python bindings. The _pyt is used to denote the python test files. We use the _pyt extension to make it easier to search for tests when running the python tests from the command line

The main module cpp file is in the root python directory. The source for each of the submodules is in a directory with the same name as the desired submodule.

Wrapping

To wrap our c++ libraries we use pybind11. This only works on linux, mac, or Visual Studio >= 2015. We enable the wrapping for our libraries by setting the IS_PYTHON_BUILD variable in the cmake and the pybind option in the conan. Both must be set to TRUE to enable the python wrapping.

Important

• Import statements at the top must include <pybind11/pybind11.h>
• Namespace decleartion must be at the top

namespace py = pybind11;

• If using boost shared pointers you must declare a pybind holder type right after namespace decleration

PYBIND11_DECLARE_HOLDER_TYPE(T, boost::shared_ptr<T>);

Setup

1. Create a directory 'python' in your source directory.
2. Add a blank __init__.py file to the above directory.
   • There must be a blank __init__.py file in all directories in the python directory.
3. Create an _py file in the python directory for the root of the python module (i.e. <module_name>_py.cpp)
4. In the file created in previous step add a PYBIND11_MODULE call like the following:

PYBIND11_MODULE(module_name, m) {
    m.doc() = "Documentation string";  // Documentation string for this module
    m.def("version", &version, "Get current version of library");

    // SubModule Definitions
    py::module SubModule1= m.def_submodule("submodule_name_1");
    initSubModule1(SubModule1);  // These functions will be shown in a later step
    py::module SubModule2= m.def_submodule("submodule_name_2");
    initSubModulef2(SubModule2);

    // You can add as many sub-modules as your module requires.
}

5. Add a folder for each submodule. (Don't forget the __init__.py file as noted above)
6. For each submodule create an _py.h and _py.cpp (i.e. submodule_py.h, submodule_py.cpp)
   • These files are used for generating the submodule and and adding it to the main module definition
7. In the above _py.h and _py.cpp define an init function for the submodule

// header
void initSubModule1(py::module &);

// cpp
void initSubmodule1(py::module &m) {
    initSubModClass1(m); // These functions are setup in a later step
    initSubModClass2(m):
}

8. For each class you want to wrap create an _py.cpp and an _pyt.py file (i.e. SubModClass1_py.cpp, SubModClass1_pyt.py)
9. Add all needed includes and code for the wrapping, and add the definition of the function to the _py.h file used for the submodule
   • NOTE: If you would like more detailed instruction on the wrapping code see either previous libraries or the pybind documentation.

PYBIND11_DECLARE_HOLDER_TYPE(T, boost::shared::_ptr<T>);
void initSubModClass1(py::module &m) {
    py::class_<__WrappedClass__, boost::shared_ptr<__WrappedClass__>>(m, "SubModClass1")
        .def(py::init(WrappedClass::New))  // Or other init function
        .def("__str__", WrappedClass::ToString, "Get class as string.")  // String function
        .def("wrapped_function", WrappedClass::Function, "Function doc string")
    ;
}

10. Write python tests for the functions you wrapped in the previous step in the _pyt file you created using the python unittest framework.

import unittest
import module_name

class TestWrappedPython(unittest.TestCase)
    def test_string_function(self):
        x = module_name.submodule_name_1.SubModClass1()
        expected_output = "expected_output string"
        self.assertEqual(expected_output, str(x))

11. Add all the .cpp and .h files, created above, to the CMakeLists.txt file in the designated locations. (The python files get added automatically by the CI)
12. Build, test, and fix as needed.

Testing

After you have wrapped a library, it is good practice to test what you have done. Make sure you have written the python test files as described in the wrapping instructions, then do the following to test your code:

1. Open a terminal in the directory your library was built to
2. Run the test command

python -m unittest discover -v -p *_pyt.py -s ../../source_dir/python

You should see output for the tests you have written printed to the console.