Questions related to Meson+Pythran

[paugier]

Happy new year Serge and @rgommers !

I'm trying to build my packages using Pythran through Transonic with Meson. I tried to follow https://pythran.readthedocs.io/en/latest/MANUAL.html#integration-into-a-python-package and what is done in Scipy. Overall we now have a working solution (described in particular in these documents: Packaging when using Transonic and Build Fluidsim), however, I'm still unsure about what should be done for few questions related to Pythran.

Warning for @rgommers: beware if you look at the ugly things we do in meson.build files with Transonic. Python source directories with their meson.build files are generated from other meson.build files. I realize that it does not follow the Meson philosophy but it works and it's convenient. Anyway, this issue is not about that.

Pythran's documentation is quite minimalist about Meson and it might be useful to add few notes.

~/.pythranrc

With Meson, Pythran is used only to produce C++ code. Hence, many of the options in the .pythranrc files are not used. Anyway, it seems reasonable that a build of a package would be reproducible and not depend at all on a ~/.pythranrc file. I don't know if it is possible to tell Pythran to not consider ~/.pythranrc. I tried with PYTHRANRC="" pythran ... but it seems that pythran is happy with this and continues to consider ~/.pythranrc.

[compiler] section

It would be nice to have a short note (or just a link) explaining how to do with meson-python what is usually done with the [compiler] section of Pythran configuration files. What is recommended? Just environment variables?

complex_hook

The only Pythran option that I really use affecting C++ code generation is complex_hook. So with Meson, we call Pythran with --config pythran.complex_hook=pythran_complex_hook and we have a Meson option pythran-complex-hook.

In https://pythran.readthedocs.io/en/latest/MANUAL.html#customizing-your-pythranrc there is

complex_hook:

Set this to True for faster and still Numpy-compliant complex multiplications. Not very portable, but generally works on Linux.

I don't know if it means that it could be reasonable to build wheels uploaded on PyPI to set pythran-complex-hook to true on Linux and false elsewhere? Also I don't know what would be the best way to do that with meson-python...

USE_XSIMD and -Ofast -march=native

It seems to me that Scipy build does not use USE_XSIMD. I wonder if it is because it would not be portable enough? However, for my case, I'd like to be able when installing from source to use this flag and also -Ofast -march=native. Moreover, it is not clear to me how this should be done with Meson-python.

[rgommers]

Happy new year @paugier!

Pythran's documentation is quite minimalist about Meson and it might be useful to add few notes.

Sure, happy to submit a PR after we finish discussing in this issue.

Anyway, it seems reasonable that a build of a package would be reproducible and not depend at all on a ~/.pythranrc file.

+1. I'm not sure if it's possible to tell Pythran to not look at ~/.pythranrc. There probably should be a CLI flag for that at least?

It would be nice to have a short note (or just a link) explaining how to do with meson-python what is usually done with the [compiler] section of Pythran configuration files. What is recommended? Just environment variables?

For compiler selection the standard CC/CXX environment variables indeed. And also CFLAGS/CXXFLAGS and LDFLAGS are honored. The other things in [compiler] look like they all belong in meson.build files.

compiler.blas should probably be ignored explicitly via -DPYTHRAN_BLAS_NONE like SciPy does by default. And arguably pythran-openblas should be removed completely from Pythran and PyPI, it's 5 years old and 20 minor releases of OpenBLAS behind. OpenBLAS 0.3.6 is very buggy and if an extension is also using numpy having a second OpebLAS copy in memory may lead to issues. It's only used for a single function, numpy.dot, with a few CBLAS functions called. And the include <cblas.h> isn't portable (e.g., for MKL it's mkl_cblas.h not cblas.h).

I don't know if it means that it could be reasonable to build wheels uploaded on PyPI to set pythran-complex-hook to true on Linux and false elsewhere? Also I don't know what would be the best way to do that with meson-python...

I have no knowledge of what complex_hook does, but shipping an sdist and wheels with meson-python shouldn't be hard. In your meson.build file, something like (adding a complete example from SciPy here, the if host_machine().system() == 'linux' is the added part here):

cmd = [pythran, '-E', '@INPUT@', '-o', '@OUTDIR@/_rbfinterp_pythran.cpp']
if host_machine().system() == 'linux'
  cmd += ['--config pythran.complex_hook=pythran_complex_hook']
endif

_rbfinterp_pythran = custom_target('_rbfinterp_pythran',
  output: ['_rbfinterp_pythran.cpp'],
  input: '_rbfinterp_pythran.py',
  command: cmd,
)

_rbfinterp_pythran = py.extension_module('_rbfinterp_pythran',
  _rbfinterp_pythran,
  cpp_args: cpp_args_pythran,
  dependencies: [pythran_dep, np_dep],
  install: true,
  subdir: 'scipy/interpolate'
)

It seems to me that Scipy build does not use USE_XSIMD. I wonder if it is because it would not be portable enough? However, for my case, I'd like to be able when installing from source to use this flag and also -Ofast -march=native. Moreover, it is not clear to me how this should be done with Meson-python.

I need to double-check this in SciPy, will circle back. Pythran is certainly looking for xsimd by default in the SciPy build, since I remember fixing a couple of issues with that.

To add the flag, it looks like you can simply add -DUSE_XSIMD to cmd in the example above? The include paths should already be okay if you copied what SciPy has in scipy/meson.build.

[paugier]

Thanks for your answers @rgommers.

Regarding USE_XSIMD, I think we need cpp_args_pythran += ['-DUSE_XSIMD'], which is not done for Scipy. @serge-sans-paille do you think -DUSE_XSIMD can be used to build wheels uploaded on PyPI?

For Fluidsim, I now use:

option(
    'native',
    type: 'boolean',
    value: false,
    description: 'Performance oriented and not portable build',
)
option(
    'use-xsimd',
    type: 'boolean',
    value: true,
    description: 'Turns on xsimd vectorization',
)
option(
    'pythran-complex-hook',
    type: 'combo',
    choices: ['os-dependent', 'true', 'false'],
    value: 'os-dependent',
    description: 'Pythran complex_hook option',
)

and (in the main meson.build file):

use_pythran = backend.contains('pythran')
if use_pythran
  incdir_numpy = run_command(
    py,
    [
      '-c',
      '''import os
import numpy as np
try:
  incdir = os.path.relpath(np.get_include())
except Exception:
  incdir = np.get_include()
print(incdir)'''
    ],
    check: true
  ).stdout().strip()

  inc_np = include_directories(incdir_numpy)
  np_dep = declare_dependency(include_directories: inc_np)

  incdir_pythran = run_command(
      py,
      [
        '-c',
        '''import os
import pythran;
incdir = os.path.dirname(pythran.__file__)
try:
  incdir = os.path.relpath(incdir)
except Exception:
  pass
print(incdir)'''
      ],
      check: true
    ).stdout().strip()

  pythran = find_program('pythran', native: true)

  cpp_args_pythran = [
    '-DENABLE_PYTHON_MODULE',
    '-D__PYTHRAN__=3',
    '-DPYTHRAN_BLAS_NONE'
  ]

  if get_option('use-xsimd') == true
    # xsimd is unvendored from pythran by conda-forge, and due to a compiler
    # activation bug the default <prefix>/include/ may not be visible (see
    # gh-15698). Hence look for xsimd explicitly.
    xsimd_dep = dependency('xsimd', required: false)
    pythran_dep = declare_dependency(
      include_directories: incdir_pythran,
      dependencies: xsimd_dep,
    )
    cpp_args_pythran += ['-DUSE_XSIMD']
  else
    pythran_dep = declare_dependency(
      include_directories: incdir_pythran,
    )
  endif

  pythran_complex_hook = get_option('pythran-complex-hook')
  if pythran_complex_hook == 'os-dependent'
    pythran_complex_hook = host_machine.system() == 'linux'
  endif

  if get_option('native')
    cpp_args_pythran += ['-march=native', '-Ofast']
  endif

endif

I wouldn't be against some helpers given by meson-python to simplify this, but I don't know if it would be possible (one can't define functions or macros in Meson).

I defined the option native because it seems to me that it's an important feature for a computational Python project to be able to be compiled in approximately two modes :

1. to build portable wheels,
2. to build for the specific CPU used for compilation (-march=native).

Is there a better way to do that with Meson and meson-python ?

[paugier]

With #2171, one can ignore ~/.pythranrc by calling Pythran in Meson files with:

_rbfinterp_pythran = custom_target('_rbfinterp_pythran',
  output: ['_rbfinterp_pythran.cpp'],
  input: '_rbfinterp_pythran.py',
  command: cmd,
  env: ['PYTHRANRC='],
)

A good point is that it can already be done (with no effect for pythran<=0.15.0).

[rgommers]

Regarding USE_XSIMD, I think we need cpp_args_pythran += ['-DUSE_XSIMD'], which is not done for Scipy. @serge-sans-paille do you think -DUSE_XSIMD can be used to build wheels uploaded on PyPI?

Ah right - I think that's the difference between using xsimd's vectorization/SIMD code paths, or the scalar/fallback paths? We certainly had build failures in SciPy on xsimd going missing in conda-forge, so it is needed.

xsimd is unvendored from pythran by conda-forge

This unvendoring was reverted ~8 months ag, as it was not a good idea: conda-forge/pythran-feedstock#76

[...] 2. to build for the specific CPU used for compilation (-march=native).

The two "build modes" make perfect sense, but I'm not sure that it needs a package-defined build option. You can simply put -march=native in CXXFLAGS or pass them with the cpp_args compiler option. So this works naturally already without custom code in your own meson.build files.

I wouldn't be against some helpers given by meson-python to simplify this, but I don't know if it would be possible (one can't define functions or macros in Meson).

For dependency detection I have the same clumsiness with dependency('numpy'). I plan to make that "just work" as a one-liner by adding a numpy.pc pkg-config file and a numpy-config script, so that there is no need to invoke Python to get at the get_include().

[rgommers]

For dependency detection I have the same clumsiness with dependency('numpy'). I plan to make that "just work" as a one-liner by adding a numpy.pc pkg-config file and a numpy-config script, so that there is no need to invoke Python to get at the get_include().

I'll note that this is done in numpy/numpy#25730 and mesonbuild/meson#12799, it should be available in Meson 1.4.0 / NumPy 2.0.0. Once that's in use without hiccups, it'd be nice to do the same for Pythran. Or do it now, since it's fairly safe (it's more or less a copy of what already worked for pybind11).

[serge-sans-paille]

Happy new year Serge and @rgommers !

I'm a bit late to the party :-)

I'm trying to build my packages using Pythran through Transonic with Meson. I tried to follow https://pythran.readthedocs.io/en/latest/MANUAL.html#integration-into-a-python-package and what is done in Scipy. Overall we now have a working solution (described in particular in these documents: Packaging when using Transonic and Build Fluidsim), however, I'm still unsure about what should be done for few questions related to Pythran.

Warning for @rgommers: beware if you look at the ugly things we do in meson.build files with Transonic. Python source directories with their meson.build files are generated from other meson.build files. I realize that it does not follow the Meson philosophy but it works and it's convenient. Anyway, this issue is not about that.

Pythran's documentation is quite minimalist about Meson and it might be useful to add few notes.

It's been improving recently (in addition to what @rgommers already wrote)

~/.pythranrc

With Meson, Pythran is used only to produce C++ code. Hence, many of the options in the .pythranrc files are not used. Anyway, it seems reasonable that a build of a package would be reproducible and not depend at all on a ~/.pythranrc file. I don't know if it is possible to tell Pythran to not consider ~/.pythranrc. I tried with PYTHRANRC="" pythran ... but it seems that pythran is happy with this and continues to consider ~/.pythranrc.

This part seems fixed:

(pythran) ssp@lakota:~/sources/pythran (fix/pythran-config)% pythran-E a.py -vv  
INFO:      sys file exists: /home/ssp/sources/pythran/pythran/pythran.cfg
INFO: platform file exists: /home/ssp/sources/pythran/pythran/pythran-linux.cfg
INFO:     user file exists: /home/ssp/.pythranrc
INFO: pythranrc section [pythran] is valid and options are correct
INFO: pythranrc section [typing] is valid and options are correct
INFO: pythranrc section [backend] is valid and options are correct
INFO: pythranrc section [compiler] is valid and options are correct
INFO: Generated C++ source file: a.cpp
(pythran) ssp@lakota:~/sources/pythran (fix/pythran-config)% PYTHRANRC= pythran -E a.py -vv 
INFO:      sys file exists: /home/ssp/sources/pythran/pythran/pythran.cfg
INFO: platform file exists: /home/ssp/sources/pythran/pythran/pythran-linux.cfg
INFO: pythranrc section [pythran] is valid and options are correct
INFO: pythranrc section [typing] is valid and options are correct
INFO: pythranrc section [backend] is valid and options are correct
INFO: pythranrc section [compiler] is valid and options are correct
INFO: Generated C++ source file: a.cpp

I agree that for reproducibility, it's better to either force no pythranrc, or use a local one, shipped with the package.

[compiler] section

It would be nice to have a short note (or just a link) explaining how to do with meson-python what is usually done with the [compiler] section of Pythran configuration files. What is recommended? Just environment variables?

That's now in the doc ;-)

complex_hook

The only Pythran option that I really use affecting C++ code generation is complex_hook. So with Meson, we call Pythran with --config pythran.complex_hook=pythran_complex_hook and we have a Meson option pythran-complex-hook.

In https://pythran.readthedocs.io/en/latest/MANUAL.html#customizing-your-pythranrc there is

complex_hook:
Set this to True for faster and still Numpy-compliant complex multiplications. Not very portable, but generally works on Linux.

I don't know if it means that it could be reasonable to build wheels uploaded on PyPI to set pythran-complex-hook to true on Linux and false elsewhere? Also I don't know what would be the best way to do that with meson-python...

I think it's fine to have different values for different environment there.

USE_XSIMD and -Ofast -march=native

It seems to me that Scipy build does not use USE_XSIMD. I wonder if it is because it would not be portable enough? However, for my case, I'd like to be able when installing from source to use this flag and also -Ofast -march=native. Moreover, it is not clear to me how this should be done with Meson-python.

Passing the option through a local pythranrc (or custom pythran flags through --config) seems fine, but please be aware that -march=native shouldn't be used to build a wheel, because if the builder supports, say, avx512 and the final user machine doesn't, this ends in a segfault. I don't know what are the minimal arch requirements for conda... maybe some flavor of SSE?