Skip to content

Home

Jump to bottom
Anders Reenberg Andersen edited this page Feb 19, 2025 · 30 revisions


MDPSolver (mdpsolver) is a Python package for large Markov Decision Processes (MDPs) with infinite horizons. This wiki includes details about the documentation of MDPSolver and other information, such as performance tests.

Overview

  • Fast solver: Our C++-based solver is substantially faster than other MDP packages available for Python, depending on the problem size and parameters. See the Performance tests section in this wiki.
  • Available on PyPI.
  • Two optimality criteria: Discounted and Average reward.
  • Three optimization algorithms: Value iteration, Policy iteration, and Modified policy iteration.
  • Three value-update methods: Standard, Gauss–Seidel, and Successive over-relaxation.
  • Uses span norm or supremum norm stopping criterion depending on the selected update method.
  • Supports sparse matrices and parallel computing.
  • Operating systems: Linux and Windows.

How to install

Linux

Install directly from PyPI with:

pip install mdpsolver

MDPSolver works out of the box on Ubuntu 22 and newer.

GLIBC not found

Some users will encounter the version 'GLIBC_2.32' not found error when attempting to import MDPSolver in Python. In this case, it might help to manually compile and replace the SO-file for the optimization module in the MDPSolver package. Here are the steps:

  1. Ensure you have python3.10-config by running sudo apt-get install python3.10-dev.
  2. Install pybind11: pip install pybind11.
  3. Clone the repository for MDPSolver and navigate to CPP_Source_Code.
  4. Assuming you have g++ installed, compile the SO-file by running:
g++ -O3 -fopenmp -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` *.cpp -o solvermodule`python3-config --extension-suffix`
  1. You should now see an SO-file solvermodule.cpython-310-x86_64-linux-gnu.so. Use pip show mdpsolver to locate the package on your machine (e.g. .local/lib/python3.10/site-packages/mdpsolver/).
  2. Navigate to the directory of MDPSolver and replace the SO-file.

Windows

Requires Visual Studio 2022 (17.9) with MSVC C++ compiler and libraries installed (see below).

Logo

After installing Visual Studio (incl. MSVC C++ compiler and libraries), install directly from PyPI with:

pip install mdpsolver

Requirements

  • Operating system: Windows or Linux.
  • Windows users must install Visual Studio 2022 incl. the MSVC C++ compiler (see the installation instructions above).
  • MDPSolver has been tested on Windows 11 and Ubuntu 22.04.5 LTS.
  • Linux users must ensure that GLIBC 2.32 or newer is installed (check your version with ldd --version).

How to cite

DOI

Clone this wiki locally