The "-march=native" flag vs. the compiler used when Python extensions are built #31228

mkoeppe · 2021-01-13T01:51:14Z

#27122 determines whether CC accepts -march=native and then adds it globally to CFLAGS in build/bin/sage-build-env.

However, Python extension modules are compiled with the compiler listed in sysconfig. On macOS with homebrew, using /usr/bin/python3, this could be clang, which does not accept -march=native.

Depends on #3113
Depends on #30725

CC: @kliem @zlscherr @jhpalmieri

Component: build: configure

Reviewer: Jonathan Kliem

Issue created by migration from https://trac.sagemath.org/ticket/31228

The text was updated successfully, but these errors were encountered:

kliem · 2021-01-13T07:29:37Z

comment:2

Do we manually set the Compiler to be different somewhere vor is this enforced by using system python? If not, is there a reasonable place to check this?

Can this happen to python modules only?

kliem · 2021-01-13T07:30:58Z

comment:3

There is CFLAGS_NON_NATIVE etc that should be used in case the compiler changes.

mkoeppe · 2021-01-13T20:39:24Z

comment:4

This is the standard behavior of python's distutils. Only Python extension modules are affected.

m4/sage_check_python_for_venv defines a macro used by build/pkgs/python3/spkg-configure.m4 to check the suitability of distutils.

These are probably the best places to put additional checks.

kliem · 2021-01-13T20:54:04Z

comment:5

I see. I can add a file compiling with -march=native in m4/sage_check_python_for_venv. In case of failure, I have to export something.

If this fails, I have to build extension modules with export CFLAGS=CFLAGS_NON_NATIVE and export CXXFLAGS=CXXFLAGS_NON_NATIVE somehow.

kliem · 2021-01-13T20:57:46Z

comment:6

Much easier would be to give completely up on march=native in this case.

mkoeppe · 2021-01-13T21:04:16Z

comment:7

Replying to @kliem:

Much easier would be to give completely up on march=native in this case.

Yes, I think that would be better. Otherwise too many spkg-install files would have to be changed. I don't see a clean way of implementing this in the helper functions sdh_pip_install and sdh_setup_bdist_wheel.

mkoeppe · 2021-01-14T04:20:26Z

Branch: u/mkoeppe/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built

sagetrac-git · 2021-01-14T04:29:23Z

Commit: 289b162

sagetrac-git · 2021-01-14T04:29:23Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`2c60025`	`m4/sage_check_python_for_venv.m4: Warn if python3 is misconfigured like it is currently in homebrew`
`ee679bd`	`build/pkgs/python3/spkg-configure.m4: Do not check sysconfig CPPFLAGS so that /usr/bin/python3 is accepted on macOS; reject broken homebrew python3 unless requested explicitly with configgure --with-python=...`
`29aee87`	`build/pkgs/python3: Update to 3.9.0rc2`
`e61d464`	`build/pkgs/python3: Update to 3.9.0`
`f8bb56d`	`build/pkgs/python3: Update to 3.9.1`
`0e3513f`	`build/pkgs/pip: Update to 20.3.3`
`41d7e7a`	`Merge branch 'u/jhpalmieri/new_wheel' of git://trac.sagemath.org/sage into t/30589/upgrade-python-3.9`
`76d5fd1`	`Merge branch 't/30589/upgrade-python-3.9' into t/31132/homebrew__unused_packages__singular__pari_______in__usr_local_leak_into_build_when_using_homebrew_s_python3`
`289b162`	`m4/sage_check_python_for_venv.m4: Factor out macros SAGE_PYTHON_DISTUTILS_C_CONFTEST, SAGE_PYTHON_DISTUTILS_CXX_CONFTEST`

mkoeppe · 2021-01-14T04:30:38Z

Dependencies: #31132

mkoeppe · 2021-01-14T04:30:38Z

comment:11

Let's do this on top of #31132, which has modified the same files.

I have added one commit that refactors m4/sage_check_python_for_venv.m4 as preparation.

sagetrac-git · 2021-01-14T04:54:51Z

Branch pushed to git repo; I updated commit sha1. New commits:

`e38057e`	`build/pkgs/python3/spkg-configure.m4: If '-march' has been set, check if it works with the compilers used for extension buildings`

sagetrac-git · 2021-01-14T04:54:51Z

Changed commit from 289b162 to e38057e

mkoeppe · 2021-01-14T04:55:14Z

comment:13

Here's an attempt

kliem · 2021-01-14T08:15:12Z

comment:14

Looks plausible, however I don't see exactly, where -march=native was added such that this command fails.

            AS_IF([CC="$CC" CXX="$CXX" conftest_venv/bin/python3 conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1 ], [

Maybe we need to add -march=native yet:

            AS_IF([CC="$CC" CXX="$CXX" CFLAGS="-march=native" CXXFLAGS="-march=native" conftest_venv/bin/python3 conftest.py --verbose build --build-base=conftest.dir >& AS_MESSAGE_LOG_FD 2>&1 ], [

But I'm just hoping they are respected.

mkoeppe · 2021-01-14T17:50:48Z

comment:15

Yes, I meant to add CFLAGS="$CFLAGS_MARCH" in these places

kliem · 2021-01-15T21:48:42Z

Author: Matthias Koeppe

kliem · 2021-01-15T21:48:42Z

Changed branch from u/mkoeppe/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built to u/gh-kliem/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built

kliem · 2021-01-15T21:48:42Z

New commits:

`3a86a18`	`add CFLAGS and CXXFLAGS`

kliem · 2021-01-15T21:48:42Z

Changed commit from e38057e to 3a86a18

mkoeppe · 2021-01-16T04:21:07Z

comment:17

On homebrew, this will have to be tested together with #31227 if a current XCode version is in use

kliem · 2021-01-16T08:22:23Z

comment:18

There is a merge conflict with #31227.

Does the problem show up during github workflows? (If so, I can fix the merge conflict and test whether the problem goes away.)

mkoeppe · 2021-01-16T17:57:47Z

comment:19

It showed up when I tried in on my local machine.
But now I think the observed failures are actually caused by some outdated code in install scripts of some packages, which explicitly set CC=clang - see #30725.

kliem · 2021-01-22T22:20:50Z

comment:20

Instead of disabling -march=native alltogether, one might get away with just disabling it in sdh_pip_install.

mkoeppe · 2021-01-23T04:38:36Z

comment:21

I prefer the easy solution as currently implemented, see comment 7 above

mkoeppe · 2021-01-23T04:39:33Z

Changed branch from u/gh-kliem/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built to u/mkoeppe/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built

sagetrac-git · 2021-01-23T04:40:01Z

Changed commit from 3a86a18 to 09b03d2

sagetrac-git · 2021-01-23T04:40:01Z

Branch pushed to git repo; I updated commit sha1. New commits:

`02c2684`	`R is already compiled without -march=native`
`24b4fcc`	`set EXTRACFLAGS instead of CFLAGS`
`44be967`	`make code a bit more stable against future changes`
`f56e65e`	`fix CFLAGS for python3/spkg-build.in by running testcflags.sh after determination of the compiler`
`09b03d2`	`Merge branch 'u/gh-kliem/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_' of git://trac.sagemath.org/sage into t/31228/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built`

mkoeppe · 2021-01-23T04:40:55Z

Changed dependencies from #31132 to #31132, #30725

kliem · 2021-01-23T10:09:16Z

comment:26

Upgrading this to blocker, because this ticket is needed to get cython installation to work again (when python uses clang instead of gcc):

#30725 comment:50

kliem · 2021-01-23T10:20:53Z

comment:27

Replying to @kliem:

New commits:

`3a86a18`	`add CFLAGS and CXXFLAGS`

I think this commit is needed, isn't it?

kliem · 2021-01-23T10:39:03Z

Reviewer: Jonathan Kliem, https://github.com/kliem/sage/pull/35/checks

kliem · 2021-01-23T10:39:03Z

comment:29

I just started github actions along with commit 3a86a18f345a1c98411aa05254bff58ac02c3180.

Please tell me, if I need to pull other tickets yet as mentioned in:

#30725 comment:42

Btw, I personally would find it useful, those tickets needed to fix github actions on top of the current beta are collected somewhere.

mkoeppe · 2021-01-23T18:35:11Z

comment:30

Replying to @kliem:

Replying to @kliem:

New commits:

3a86a18 add CFLAGS and CXXFLAGS

I think this commit is needed, isn't it?

Right, something like this. I'll try to dig out the commits that I was using

mkoeppe · 2021-01-23T20:24:55Z

comment:31

I have pushed the branch to #31227. Maybe we can skip the present ticket and do everything in #31227.

mkoeppe · 2021-01-25T19:21:54Z

Changed author from Matthias Koeppe to none

kliem · 2021-01-25T21:28:27Z

Changed branch from u/mkoeppe/the___march_native__flag_vs__the_compiler_used_when_python_extensions_are_built to none

kliem · 2021-01-25T21:28:27Z

Changed dependencies from #31132, #30725 to #3113, #30725

kliem · 2021-01-25T21:28:27Z

Changed reviewer from Jonathan Kliem, https://github.com/kliem/sage/pull/35/checks to Jonathan Kliem

kliem · 2021-01-25T21:28:27Z

Changed commit from 09b03d2 to none

kliem · 2021-01-25T21:28:27Z

comment:33

Agreed.

mkoeppe added this to the sage-9.3 milestone Jan 13, 2021

mkoeppe added c: build: configure labels Jan 13, 2021

mkoeppe added p: critical / 2 and removed p: major / 3 labels Jan 14, 2021

mkoeppe removed the p: critical / 2 label Jan 16, 2021

mkoeppe added the s: needs review label Jan 23, 2021

kliem added p: blocker / 1 and removed p: major / 3 labels Jan 23, 2021

kliem added s: needs work and removed s: needs review labels Jan 23, 2021

mkoeppe removed this from the sage-9.3 milestone Jan 25, 2021

mkoeppe added s: needs review and removed s: needs work labels Jan 25, 2021

kliem added s: positive review and removed s: needs review labels Jan 25, 2021

vbraun removed the s: positive review label Mar 26, 2021

vbraun closed this as completed Mar 26, 2021

The "-march=native" flag vs. the compiler used when Python extensions are built #31228

The "-march=native" flag vs. the compiler used when Python extensions are built #31228

Comments

mkoeppe commented Jan 13, 2021

kliem commented Jan 13, 2021

kliem commented Jan 13, 2021

mkoeppe commented Jan 13, 2021

kliem commented Jan 13, 2021

kliem commented Jan 13, 2021

mkoeppe commented Jan 13, 2021

mkoeppe commented Jan 14, 2021

sagetrac-git mannequin commented Jan 14, 2021

sagetrac-git mannequin commented Jan 14, 2021

mkoeppe commented Jan 14, 2021

mkoeppe commented Jan 14, 2021

sagetrac-git mannequin commented Jan 14, 2021

sagetrac-git mannequin commented Jan 14, 2021

mkoeppe commented Jan 14, 2021

kliem commented Jan 14, 2021

mkoeppe commented Jan 14, 2021

kliem commented Jan 15, 2021

kliem commented Jan 15, 2021

kliem commented Jan 15, 2021

kliem commented Jan 15, 2021

mkoeppe commented Jan 16, 2021

kliem commented Jan 16, 2021

mkoeppe commented Jan 16, 2021

kliem commented Jan 22, 2021

mkoeppe commented Jan 23, 2021

mkoeppe commented Jan 23, 2021

sagetrac-git mannequin commented Jan 23, 2021

sagetrac-git mannequin commented Jan 23, 2021

mkoeppe commented Jan 23, 2021

kliem commented Jan 23, 2021

kliem commented Jan 23, 2021

kliem commented Jan 23, 2021

kliem commented Jan 23, 2021

mkoeppe commented Jan 23, 2021

mkoeppe commented Jan 23, 2021

mkoeppe commented Jan 25, 2021

kliem commented Jan 25, 2021

kliem commented Jan 25, 2021

kliem commented Jan 25, 2021

kliem commented Jan 25, 2021

kliem commented Jan 25, 2021