Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiled backends integration #835

Merged
merged 89 commits into from
Jul 22, 2022
Merged
Show file tree
Hide file tree
Changes from 80 commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
6bad5c4
compiled backends code
petiaccja Jun 20, 2022
77c2075
tests for compiled backends
petiaccja Jun 20, 2022
da31c54
rebased to Rico's backend cleanup
petiaccja Jul 4, 2022
217f6b1
merged online compilation into Rico's backend structure
petiaccja Jul 6, 2022
5b01ca2
renamed test folder to match backend's structure
petiaccja Jul 6, 2022
1307bad
set up C++ dev tools and libs
petiaccja Jul 6, 2022
e9740b1
fixed style and quality problems
petiaccja Jul 6, 2022
4bb8b4d
Merge remote-tracking branch 'origin/functional' into compiled-backen…
petiaccja Jul 7, 2022
7863592
fixed problems after merge
petiaccja Jul 7, 2022
2c05f7b
added parametric tests and fixed issues
petiaccja Jul 7, 2022
016b914
parametrized simple tests
petiaccja Jul 7, 2022
3c91964
fixed qa
petiaccja Jul 7, 2022
6c4dc2e
renamed modules -> importer
petiaccja Jul 8, 2022
2c8df46
uses compiled module suffix from importlib; uses specs for importing …
petiaccja Jul 8, 2022
4395143
removed formatting from ir
petiaccja Jul 8, 2022
d97e61a
Merge remote-tracking branch 'origin/functional' into compiled-backen…
petiaccja Jul 11, 2022
5aec53a
use pybind11 from pip
petiaccja Jul 11, 2022
79d06c6
remove pybind11 from ci yaml
petiaccja Jul 11, 2022
1629234
fixed tests and formatting
petiaccja Jul 11, 2022
cf87aa7
simplified parameter creation helpers
petiaccja Jul 11, 2022
3184f80
added a sid expr node and moved stencil argument rendering into code …
petiaccja Jul 11, 2022
d663b65
links to GridTools::fn_naive and removed all explicit references to o…
petiaccja Jul 11, 2022
5a1ace1
require c++17 on target instead of globally
petiaccja Jul 11, 2022
4ec81b3
removed threads library requirement from cmake
petiaccja Jul 11, 2022
aad9b05
fixed cartesian offsets
petiaccja Jul 11, 2022
8b3114c
removed questionable line of test
petiaccja Jul 11, 2022
e4424e9
parametrize fieldview tests and skip unsupported
Jul 11, 2022
93af467
Merge branch 'cbi-plus-tests' into compiled-backends-integration
Jul 11, 2022
aee5887
added license headers for the entire fencil_processors folder
petiaccja Jul 12, 2022
5fc0f98
typo in type annotations
petiaccja Jul 12, 2022
ece2641
using match statement in cache instead of dict
petiaccja Jul 12, 2022
4c19f07
simplify pytest.skip calls, run pre-commit
Jul 13, 2022
e6d0871
adr parts
petiaccja Jul 13, 2022
3c5a8b1
stage
petiaccja Jul 14, 2022
ee0059d
few words about limitations
petiaccja Jul 14, 2022
ccacd74
Update ADR
DropD Jul 14, 2022
827ce57
use ninja and cmake from pip
petiaccja Jul 15, 2022
887020b
extended caching to consider all module parameters
petiaccja Jul 18, 2022
b5fbaea
Update src/functional/fencil_processors/cpp.py
petiaccja Jul 19, 2022
de16028
renamed some things
petiaccja Jul 19, 2022
8a220b8
some missing return types
petiaccja Jul 19, 2022
0a55dea
update ADR based on review comments
Jul 19, 2022
c3f250c
improve ADR
Jul 19, 2022
01c3a93
add docstrings to cbi caching
Jul 19, 2022
ea121d0
cbi: add importer docstring
Jul 19, 2022
05daec4
rename var
petiaccja Jul 19, 2022
6389e7b
some doc comments
petiaccja Jul 19, 2022
3f9a965
fix tempdir
petiaccja Jul 20, 2022
f1367a6
replaced deprecated typing.Callable with collections.abc.Callable
petiaccja Jul 20, 2022
27e0cfa
grammar / typo
petiaccja Jul 20, 2022
494c9e0
naming conventions
petiaccja Jul 20, 2022
bc34df0
enforce keyword arguments for optionals
petiaccja Jul 20, 2022
c2ad1dd
using as_jinja for JinjaTemplate for consistency
petiaccja Jul 20, 2022
3204d7f
cbi: eve.codegen import conventions
Jul 20, 2022
03179ee
Merge remote-tracking branch 'upstream/compiled-backends-integration'…
Jul 20, 2022
0322f84
cbi: bindings docstrings
Jul 20, 2022
48c9baa
cbi: builders.cpp.build docstrings
Jul 20, 2022
0fe9396
cbi: ADR and some todos
Jul 20, 2022
16de03c
cbi: simplify importer
Jul 20, 2022
63c59f0
Update src/functional/fencil_processors/codegens/gtfn/gtfn_module.py
DropD Jul 20, 2022
c9d7d32
avoid making numpy copies
petiaccja Jul 20, 2022
b5d550e
cleaned up cmakeproject
petiaccja Jul 20, 2022
af2792c
redo ugly method to fix stuff but this could indeed be improved
petiaccja Jul 20, 2022
5004ed0
cbi: revert earlier erroneous importer change
Jul 20, 2022
83589e1
Merge remote-tracking branch 'upstream/compiled-backends-integration'…
Jul 20, 2022
20fc3b1
cbi: small formatting change
Jul 20, 2022
8e292d3
more doc
petiaccja Jul 20, 2022
5051334
remove unnecessary functions
petiaccja Jul 20, 2022
95f2f75
further simplification
petiaccja Jul 20, 2022
2115514
changed source modules to use numpy.dtype instead of raw types
petiaccja Jul 20, 2022
a4beee1
bugfix
petiaccja Jul 21, 2022
8d4b9a2
delete init.py from tests
petiaccja Jul 21, 2022
a63070a
simplified code with fstrings
petiaccja Jul 21, 2022
818b1a5
froze dataclasses for source modules
petiaccja Jul 21, 2022
5fa4dff
formatting
petiaccja Jul 21, 2022
8ba269e
simplified binding code gen
petiaccja Jul 21, 2022
50a0111
cleaned up bindings ast
petiaccja Jul 21, 2022
5eb5905
Merge remote-tracking branch 'origin/functional' into compiled-backen…
petiaccja Jul 21, 2022
a896ede
fixed tests
petiaccja Jul 21, 2022
ae9a2a5
minor spelling changes to adr
petiaccja Jul 21, 2022
ca012a3
typo
petiaccja Jul 22, 2022
d85cdd1
typo
petiaccja Jul 22, 2022
b1779e0
minor fixes
petiaccja Jul 22, 2022
505e192
adding CMakeProject design considerations to ADR
Jul 22, 2022
2e3ddc2
cpp->python type mapping global & MappingProxyType
petiaccja Jul 22, 2022
2e43a85
formatting
petiaccja Jul 22, 2022
c46611c
wording
petiaccja Jul 22, 2022
6e05a26
wording & clarification
petiaccja Jul 22, 2022
cc28b32
delete init.py from tests
petiaccja Jul 22, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/tox.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ jobs:
- name: Install python dependencies
run: |
python -m pip install --upgrade pip setuptools wheel
- name: Install C++ dependencies
run: |
sudo apt install libboost-all-dev
- name: Test with tox
run: |
pyversion_no_dot="${{ matrix.python-version }}"
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ __pycache__/

# Distribution / packaging
.Python
build*/
build/
develop-eggs/
dist/
downloads/
Expand Down
118 changes: 118 additions & 0 deletions docs/functional/architecture/0009-Compiled-Backend-Integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# Integration of compiled backends

- **Status**: valid
- **Authors**: Peter Kardos (@petiaccja), Rico Häuselmann (@DropD)
- **Created**: 2022-07-13
- **Updated**: 2022-07-13

Summary of the key design choices made for the live execution of generated C++ (and other compiled) code.

## Context

As per the current state, gt4py can execute iterator IR by generating Python code from it which is loaded back as a module. Additionally, gt4py can also emit equivalent C++ code from iterator IR. The integration of the compiled backends focuses on compiling the emitted C++ code and calling it from Python, effectively executing iterator IR live by translating it to machine code.

The process of executing ITIR this way consists of the following steps:
1. Generate C++ code equivalent to ITIR (*fencil code*)
2. Generate C++ code that wraps the fencil code and can be accessed from Python (*binding code*)
3. Compile the fencil and binding codes into machine code (a dynamically linked library)
4. Load the dynamic library into Python and extract the interfaces as Python objects
5. Call the interfaces from Python with the fencil's arguments

Step 1 is the core responsibility of a compiled backend. For Interoperability between steps, the `fencil_processors.source_modules` subpackage provides utilities and data structures which should simplify implementation and guide design, without restricting the backend implementer (they are opt-in). Steps 2-4 should rely on `fencil_processors.builders` with little code required on the backend side. For backends where additional functionality is required, it is expected that the missing functionality be implemented inside those subpackages. Step 5 should typically simply relay the fencil arguments to the result of step 4 without additional code required.
petiaccja marked this conversation as resolved.
Show resolved Hide resolved

The desired pipeline architecture for these steps is made easy to achieve by the provided library, on a per-backend level. The clear separation of the steps is not yet enforced, however, and neither is it completely implemented for the `gtfn` backend yet.

For reference, steps 2-4 are currently encoded in `fencil_processors.builders.callable.create_callable`, while examples for step 1 and 5 can be found at `fencil_processors.codegens.gtfn.gtfn_backend.generate` and `fencil_processors.runners.gtfn.run_gtfn` (which also encompasses all the other steps).

## Python bindings for C++ code

### Alternatives

The most popular libraries to make C++ code accessible from Python are:

- **pybind11**
- Boost.Python
- Python/C & ctypes

### Decision

Python/C is a good choice if we want to have a C interface to the compiled code, which can then be called not only form Python, but from pretty much any other language. On the other hand, having to manually handle C++ types and exceptions makes this approach much more complicated.
petiaccja marked this conversation as resolved.
Show resolved Hide resolved

Boost.Python and pybind11 are very similar. Since boost is not a requirement for the bindings themselves (though it is for all current C++ fencil code), it makes sense to use the much more lightweight and perhaps more modern alternative of pybind11. Unlike Python/C, pybind11 handles STL containers and name mangling out of the box, and it also has a much more concise interface.

## Interface between binding and fencil code

### Alternatives

- Slim interface: requires implementing a tailored binding code generator for every fencil code generator
- **Universal interface**: requires all fencil code generators of the same technology (i.e. all C++ generators) to have the same interface. This allows one to write a single binding generator for C++ that can be paired with any fencil code generator

### Decision

We chose the universal interface for the following advantages:
- Promotes code reuse: slim bindings for different C++ backends would share a large fraction of their code
- Decoupling fencil generators from binding generators allows testing the two components separately
- Forces the separation of responsibilities between binding and fencil code generation

However there are some downsides and implementation issues:
- Although the proper architecture is in place, the current implementation still leaves the fencil and binding code somewhat intertwined
- Currently only two C++ backends are foreseen: GridTools CPU and GridTools CUDA. These backends have close to or completely identical C++ code. This situation might make code reuse less significant.


## Connection to FieldView frontend

We decided to keep compiled backend completely independent of the fields view frontend. The input the compiled backends is thus iterator IR and additional information necessary for code generation.
petiaccja marked this conversation as resolved.
Show resolved Hide resolved

Reasons:
- There may be multiple frontends, but all of them would emit iterator IR and therefore would be compatible with compiled backends out of the box
- The FieldView frontend must import compiled backends. If compiled backends were aware of FieldView features, that would lead to a circular dependency between the frontend and backend.
- The work to support starting from iterator IR directly would have to be done in any case.


## Limitations

The main goal of this project is to implement the complete pipeline from FieldView to machine code and demonstrate that it's working. To keep the scope of the project reasonable, feature completeness is not targeted.

### Desired Architecture

As stated above, the architectural goal that each step of the compilation process with a potentially useful output should stand alone and be accessible through a unified interface is not quite reached yet. These useful outputs are generated code, a build system project with generated code and (optionally) python bindings code, a python extension module containing the callable fencil (or the callable fencil from it).
egparedes marked this conversation as resolved.
Show resolved Hide resolved

The `FencilExecutor` provided by `gtfn` should be refactored to run the necessary steps in a declarative way, so no other logic (which should really be in one of the steps) can be introduced.

### Splitting existing fencil and binding code generators

The "roundtrip" backend could benefit from being split into a Python fencil generator and a small binding generator, which would merely write out the file and load it back live.

This is out of the scope of this project, but should be done in the future. Additional backends, wherever possible, should follow the separated pattern for clarity.

### Fencil argument types

In this implementation, only fencil having scalar and field arguments can be executed with compiled backends. Constant fields and index fields are to be added later.

### GridTools CUDA support

To keep things simple, GridTools CUDA is not supported in this first iteration.

To add support, we have to decide on the interface that is exposed to the user:
1. There are two completely separate backends for CPU and CUDA
2. There is a single backend and the device is selected by a flag

The existing code can be extended for either options in the future.

### Unstructured grids

This implementation only supports Cartesian grids, again, to keep things simple. Unstructured grids are to be added as soon as possible, however, that requires more design when it comes to passing the connectivities from Python to C++.

### Library dependencies

Compiled backends may generate code which depends on libraries and tools written in the language in which the backend generates code. There are three different categories currently, each occurring in the `gtfn` backend. They are libraries and tools, which

1. can be installed with `pip` (from `PyPI` or another source) automatically.
2. can not be installed with `pip` and not commonly found on HPC machines.
3. libraries and tools which are left to the user to install and make discoverable: `pybind11`, C++ compilers

Category 1 are made dependencies of `GT4Py`. Examples include `pybind11`, `cmake`, `ninja`.

The core library the backend is based on typically falls into Category 2. The only one currently is `GridTools`, which is downloaded and installed as part of the build process for each fencil. This is not part of the design but an implementation detail which should be changed (for example by moving `GridTools` into Category 1)

Category 3 contains compilers for the backend's language and libraries like `boost`, `mpi`, `CUDA` etc. They are currently left up to the user to deal with. In the long run user experience could be improved by providing a package in an appropriate package manager, which resolves these dependencies automatically.
2 changes: 2 additions & 0 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ install_requires =
black>=22.3
boltons>=20.0
click>=7.1
cmake>=3.22
cytoolz>=0.11
deepdiff>=5.8
devtools>=0.5
Expand All @@ -52,6 +53,7 @@ install_requires =
lark>=1.1.2
mako>=1.1
networkx>=2.4
ninja>=1.10
numpy>=1.21
packaging>=20.0
pybind11>=2.5
Expand Down
13 changes: 13 additions & 0 deletions src/functional/fencil_processors/builders/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# GT4Py Project - GridTools Framework
#
# Copyright (c) 2014-2022, ETH Zurich
# All rights reserved.
#
# This file is part of the GT4Py project and the GridTools framework.
# GT4Py is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or any later
# version. See the LICENSE.txt file at the top-level directory of this
# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
#
# SPDX-License-Identifier: GPL-3.0-or-later
90 changes: 90 additions & 0 deletions src/functional/fencil_processors/builders/cache.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# GT4Py Project - GridTools Framework
#
# Copyright (c) 2014-2022, ETH Zurich
# All rights reserved.
#
# This file is part of the GT4Py project and the GridTools framework.
# GT4Py is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or any later
# version. See the LICENSE.txt file at the top-level directory of this
# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
#
# SPDX-License-Identifier: GPL-3.0-or-later
"""Caching for compiled backend artifacts."""


import enum
import hashlib
import pathlib
import tempfile

from functional.fencil_processors import source_modules


class Strategy(enum.Enum):
SESSION = 1
PERSISTENT = 2


_session_cache_dir = tempfile.TemporaryDirectory(prefix="gt4py_session_")

_session_cache_dir_path = pathlib.Path(_session_cache_dir.name)
_persistent_cache_dir_path = pathlib.Path(tempfile.gettempdir()) / "gt4py_cache"


def _serialize_param(
parameter: source_modules.ScalarParameter | source_modules.BufferParameter,
) -> str:
if isinstance(parameter, source_modules.ScalarParameter):
return f"{parameter.name}: {str(parameter.scalar_type)}"
elif isinstance(parameter, source_modules.BufferParameter):
return f"{parameter.name}: {str(parameter.scalar_type)}<{', '.join(parameter.dimensions)}>"
raise ValueError("Invalid parameter type. This is a bug.")


def _serialize_library_dependency(dependency: source_modules.LibraryDependency) -> str:
return f"{dependency.name}/{dependency.version}"


def _serialize_module(module: source_modules.SourceModule) -> str:
parameters = [_serialize_param(param) for param in module.entry_point.parameters]
dependencies = [_serialize_library_dependency(dep) for dep in module.library_deps]
return f"""\
language: {module.language}
name: {module.entry_point.name}
params: {', '.join(parameters)}
deps: {', '.join(dependencies)}
src: {module.source_code}\
"""


def _cache_folder_name(module: source_modules.SourceModule) -> str:
serialized = _serialize_module(module)
fingerprint = hashlib.sha256(serialized.encode(encoding="utf-8"))
fingerprint_hex_str = fingerprint.hexdigest()
return module.entry_point.name + "_" + fingerprint_hex_str


def get_cache_folder(module: source_modules.SourceModule, strategy: Strategy) -> pathlib.Path:
"""
Construct the path to where the build system project artifact of a source module should be cached.

The returned path points to an existing folder in all cases.
"""
folder_name = _cache_folder_name(module)

match strategy:
case Strategy.SESSION:
base_path = _session_cache_dir_path
case Strategy.PERSISTENT:
base_path = _persistent_cache_dir_path
case _:
raise ValueError("Unsupported caching strategy.")

base_path.mkdir(exist_ok=True)

complete_path = base_path / folder_name
complete_path.mkdir(exist_ok=True)

return complete_path
19 changes: 19 additions & 0 deletions src/functional/fencil_processors/builders/cpp/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# GT4Py Project - GridTools Framework
#
# Copyright (c) 2014-2022, ETH Zurich
# All rights reserved.
#
# This file is part of the GT4Py project and the GridTools framework.
# GT4Py is free software: you can redistribute it and/or modify it under
# the terms of the GNU General Public License as published by the
# Free Software Foundation, either version 3 of the License, or any later
# version. See the LICENSE.txt file at the top-level directory of this
# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
#
# SPDX-License-Identifier: GPL-3.0-or-later


from .callable import create_callable


__all__ = ["create_callable"]
Loading