Refactor Python bindings

[stubbiali]

Profound restructuring and refining of the Python bindings. Since this effort is driven by the needs of the FVM model, the focus has primarily been on structured grids so far.

• The Python and C++ files are kept in separate directories, respectively bindings/python/src/ghex and bindings/python/src/_pyghex;
• The Python package is renamed from pyghex to ghex, and all package-level configurations are dumped into pyproject.toml;
• Split Python code into modules to reflect the file structure of C++ source directory;
• Switch build backend from scikit-build-core to setuptools.build_meta; this is mainly motivated by the flexibility provided by the latter, which allows to customize the build process using environment variables and so allows to e.g. enable GPU support and specify the transport backend;
• Fix check on the memory layout of the buffer to exchange;
• Add support for HIP/ROCm by supporting the __hip_array_interface__ (introduced by Python: Add support for HIP/ROCm buffers GridTools/gridtools#1759 and feature[cartesian]: Add support for HIP/ROCm GridTools/gt4py#1278);
• Fix compilation on NVIDIA GPUs;
• Adapt unit tests;
• Apply the black formatter to all Python source files and add annotations;
• Minor bug-fix in clang-format config file.

The code has been tested on Mac (M2 processor), Piz Daint and LUMI.

[boeschf]

Hi @stubbiali ! Thanks for all the work - looks good! I'll try to give you extended feedback today or tomorrow. Could you maybe merge with master and resolve conflicts?

[boeschf]

allows to customize the build process using environment variables and so allows to e.g. enable GPU support and specify the transport backend;

Is there any other/additional way to customize the build process? Environment variables are fine, but maybe command line arguments would be more ergonomic/more explicit.

[boeschf]

A small thing that rubs me the wrong way is that the build directory is created in the source tree next to setup.py (build_dir = os.path.abspath(os.path.join(this_dir, "build"))) - do you think we can change this? Not sure where it should be, maybe in tmp or somewhere else (or be configurable)?

[stubbiali]

Is there any other/additional way to customize the build process? Environment variables are fine, but maybe command line arguments would be more ergonomic/more explicit.

I tend to agree that command line arguments would be more elegant. The only way I know to accomplish this would be through config settings. To be honest, I've never used this feature, but to me it doesn't look like it makes the customization of the build process either more explicit or easier to read. So I would stick to environment variables, which is a more common approach.

[stubbiali]

A small thing that rubs me the wrong way is that the build directory is created in the source tree next to setup.py (build_dir = os.path.abspath(os.path.join(this_dir, "build"))) - do you think we can change this? Not sure where it should be, maybe in tmp or somewhere else (or be configurable)?

We could use a temporary folder as the build directory with build_dir = tempfile.mkdtemp(). Configuration via env variable is also possible, e.g. the build directory could be created under GHEX_BUILD_ROOT, rather than in the system-wide temporary directory.

[boeschf]

closing this as it is superseded by #156