macOS: spkg-install scripts that force use of clang conflict with '-march=native' #30725

mkoeppe · 2020-10-05T02:07:31Z

Several packages force the use of clang because they cannot be built with a real gcc due to the use of Apple language extensions in system header files.

For these packages, the use of -march=native introduced in #27122 causes build errors because clang does not understand these flags (even though /usr/bin/gcc does).

Tickets where code setting the compiler was explicitly introduced: #22999 (cmake), ....

$ git grep MACOSX_VERSION
build/pkgs/curl/spkg-install.in:        echo "OS X 10.$[$MACOSX_VERSION-4] Building with clang and --with-darwinssl."
build/pkgs/psutil/spkg-install.in:    echo "OS X 10.$[$MACOSX_VERSION-4] Building with clang."
build/pkgs/python3/spkg-build.in:        echo "OS X 10.$[$MACOSX_VERSION-4] Building with clang."
build/pkgs/r/spkg-install.in:        echo "OS X 10.$[$MACOSX_VERSION-4] Building with clang."

To test:

tox -e local-homebrew-macos-minimal
tox -e local-homebrew-macos-minimal -- cmake curl gmp
tox -e local-homebrew-macos-minimal -- ptest

To test with a real gcc (using a gcc-10 installed from homebrew):

tox -e local-direct-gcc_10 -- cmake curl gmp python3 psutil r

CC: @jhpalmieri @dimpase @kliem @zlscherr @kiwifb @seblabbe

Component: porting

Author: Matthias Koeppe, Jonathan Kliem

Branch/Commit: f56e65e

Reviewer: Jonathan Kliem, Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/30725

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-01-16T18:02:40Z

Dependencies: #31186

mkoeppe · 2021-01-18T01:15:30Z

Changed dependencies from #31186 to none

mkoeppe · 2021-01-18T01:43:10Z

Branch: u/mkoeppe/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_

kiwifb · 2021-01-18T01:47:04Z

comment:8

Those bits should have gone when I "normalised" clang for building sage on OS X. We were probably planing to remove them at a later stage and forgot.

Well, the opportunity is clearly now.

New commits:

`1cb38cd`	`tox.ini: Add configuration factor gcc_10`
`f190f51`	`build/pkgs/cmake/spkg-install.in: Do not explicitly use clang on macos`
`0c89bc0`	`build/pkgs/{gmp,mpir}/spkg-install.in: Do not override compiler set explicitly to clang`
`178a490`	`build/pkgs/pari/spkg-install.in: Do not recompute MACOSX_VERSION`
`13ef685`	`build/pkgs/python3/spkg-build.in: Do not explicitly use clang on macos`
`2a5e7db`	`build/pkgs/r/spkg-install.in: Do not explicitly use clang on macos`
`c187d65`	`build/pkgs/psutil/spkg-install.in: Do not explicitly use clang on macos`
`3fd3388`	`build/pkgs/curl/spkg-install.in: Do not explicitly use clang on macos`

kiwifb · 2021-01-18T01:47:04Z

Commit: 3fd3388

mkoeppe · 2021-01-18T01:58:50Z

comment:10

Also several of these packages have been updated since and are able to build with an actual gcc on macOS. We have tickets for some packages where this is still broken: #29613 (givaro)

sagetrac-git · 2021-01-18T05:26:51Z

Changed commit from 3fd3388 to 2a39819

sagetrac-git · 2021-01-18T05:26:51Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`5bc544c`	`build/pkgs/cmake/spkg-install.in: Do not explicitly use clang on macos`
`da4ea53`	`build/pkgs/{gmp,mpir}/spkg-install.in: Do not override compiler set explicitly to clang`
`05b06b2`	`build/pkgs/pari/spkg-install.in: Do not recompute MACOSX_VERSION`
`e796b8a`	`build/pkgs/python3/spkg-build.in: Do not explicitly use clang on macos`
`5895d23`	`build/pkgs/r/spkg-install.in: Do not explicitly use clang on macos`
`350bc01`	`build/pkgs/psutil/spkg-install.in: Do not explicitly use clang on macos`
`2a39819`	`build/pkgs/curl/spkg-install.in: Do not explicitly use clang on macos`

mkoeppe · 2021-01-18T05:27:20Z

Author: Matthias Koeppe

kliem · 2021-01-18T06:48:32Z

Reviewer: https://github.com/kliem/sage/pull/34/checks

mkoeppe · 2021-01-18T07:01:35Z

comment:15

I have tested the configuration in the ticket description successfully.

But testing with the gcc-10 from homebrew, I am still running into issues with macOS header files that are not compatible with gcc.

  [cmake-3.18.2] installing. Log file: /Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/logs/pkgs/cmake-3.18.2.log
  [cmake-3.18.2] error installing, exit status 1. End of log file:
  [cmake-3.18.2]   /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/System/Library/Frameworks/ImageIO.framework/Headers/CGImageAnimation.h:42:15: error: expected unqualified-id before '^' token
  [cmake-3.18.2]      42 | typedef void (^CGImageSourceAnimationBlock)(size_t index, CGImageRef image, bool* stop);
  [cmake-3.18.2]         |               ^

kliem · 2021-01-18T07:03:46Z

comment:16

Btw, to prevent breackage because of -march=native, one can also export CFLAGS=CFLAGS_NON_NATIVE etc. right after setting the compiler to clang.

mkoeppe · 2021-01-18T07:08:20Z

comment:17

Yes, if simple fixes such as package upgrades don't fix it, that's what we should do.

mkoeppe · 2021-01-18T07:50:50Z

comment:18

Upgrade of cmake to 3.19.3 (#31258) does not fix this

dimpase · 2021-01-18T16:39:04Z

comment:19

I don't understand - I presume one still cannot build all of Sage on macOS using "real" gcc.

Why is this a blocker?

mkoeppe · 2021-01-18T17:32:24Z

comment:20

Replying to @dimpase:

I don't understand - I presume one still cannot build all of Sage on macOS using "real" gcc.

Yes, this is what the experiment with cmake has just revealed.

Why is this a blocker?

When Apple's compiler is invoked as clang instead of gcc, it does not understand the same flags. So #27122 leads to build errors.

mkoeppe · 2021-01-19T20:40:13Z

Reviewer: https://github.com/mkoeppe/sage/actions/runs/496953211, https://github.com/mkoeppe/sage/actions/runs/496953208, ...

kliem · 2021-01-20T10:16:17Z

Changed commit from 9b0eb15 to f56e65e

kliem · 2021-01-20T10:16:17Z

Changed branch from u/mkoeppe/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_ to u/gh-kliem/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_

kliem · 2021-01-20T10:16:17Z

comment:35

R does not use -march=native anyway.
python/spkg-build.in is set up a bit different. $CFLAGS contains other stuff and $EXTRA_CFLAGS is contains the old $CFLAGS, so we need to set $EXTRA_CFLAGS.

Anyway, I think the main problem is that we run testcfags.sh before setting the compiler.
We might as well run this for $CFLAGS as well after possibly changing the compiler.

New commits:

`02c2684`	`R is already compiled without -march=native`
`24b4fcc`	`set EXTRACFLAGS instead of CFLAGS`
`44be967`	`make code a bit more stable against future changes`
`f56e65e`	`fix CFLAGS for python3/spkg-build.in by running testcflags.sh after determination of the compiler`

kliem · 2021-01-20T10:16:17Z

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/496953211, https://github.com/mkoeppe/sage/actions/runs/496953208, ... to https://github.com/mkoeppe/sage/actions/runs/496953211, https://github.com/mkoeppe/sage/actions/runs/496953208, Jonathan Kliem

kliem · 2021-01-20T10:24:57Z

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/496953211, https://github.com/mkoeppe/sage/actions/runs/496953208, Jonathan Kliem to https://github.com/mkoeppe/sage/actions/runs/496953211, https://github.com/mkoeppe/sage/actions/runs/496953208, Jonathan Kliem, https://github.com/kliem/sage/actions/runs/498284561

mkoeppe · 2021-01-20T20:50:48Z

comment:37

Replying to @kliem:

R does not use -march=native anyway.

... is it ignoring CFLAGS?

kliem · 2021-01-20T20:59:19Z

comment:38

No, it already uses CFLAGS_NON_NATIVE.

 10 # #29170: Compilation errors caused by a silently failing configure check
 11 # "for type of 'hidden' Fortran character lengths"
 12 # on ubuntu-bionic-minimal, ubuntu-eoan/focal-minimal, debian-buster/bullseye/sid-minimal,
 13 # linuxmint-19.3-minimal, archlinux-latest-minimal
 14 CFLAGS="$CFLAGS_NON_NATIVE -fPIC"
 15 FCFLAGS="$FCFLAGS_NON_NATIVE -fPIC"

mkoeppe · 2021-01-20T21:00:28Z

comment:39

Ah, OK, thanks

kliem · 2021-01-20T21:00:45Z

comment:40

R was one of the things that crashed or didn't build with -march=native (don't remember), so I disalbed it in #27122.

kliem · 2021-01-21T07:36:18Z

comment:41

My tests finished. Not very sucessfull though. Looks like zlib failed for many mac runs.

Also ubuntu-bionic-minimal and debian-buster-minimal failed because somehow ssl wasn't available and the pytest failed (or the server did not reply).

Anyway. There are several different ways to fix python3/spkg-build.in of course.

mkoeppe · 2021-01-21T22:58:14Z

comment:42

Replying to @kliem:

My tests finished. Not very sucessfull though. Looks like zlib failed for many mac runs.

Also ubuntu-bionic-minimal and debian-buster-minimal failed because somehow ssl wasn't available and the pytest failed (or the server did not reply).

One needs to merge a few tickets that fix up the GH Actions runs.

mkoeppe · 2021-01-21T22:59:32Z

comment:43

Replying to @kliem:

There are several different ways to fix python3/spkg-build.in of course.

Are you satisfied with the version that you pushed?

mkoeppe · 2021-01-21T23:02:22Z

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/496953211, https://github.com/mkoeppe/sage/actions/runs/496953208, Jonathan Kliem, https://github.com/kliem/sage/actions/runs/498284561 to https://github.com/mkoeppe/sage/actions/runs/502363182, Jonathan Kliem

kliem · 2021-01-22T06:54:29Z

comment:45

Replying to @mkoeppe:

Replying to @kliem:

There are several different ways to fix python3/spkg-build.in of course.

Are you satisfied with the version that you pushed?

Yes.

kliem · 2021-01-22T06:54:57Z

comment:46

And I'm also happy with your changes.

I'm just waiting on an okay, that this fixes things.

zlscherr · 2021-01-22T22:02:06Z

comment:47

Is this supposed to fix issues around using /usr/bin/python3 on Big Sur? With this ticket if I configure with-python=/usr/bin/python3 and then do make Cython I still get

gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -O2 -g -march=native -I/Users/zscherr/sage/develop/local/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c /private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-rpwm0h2y/Cython/Plex/Scanners.c -o build/temp.macosx-10.14.6-x86_64-3.8/private/var/folders/5f/z870t4kd5dlctkcrs984nh8r0000gn/T/pip-req-build-rpwm0h2y/Cython/Plex/Scanners.o
  clang: error: the clang compiler does not support '-march=native'
  error: command 'gcc' failed with exit status 1

kliem · 2021-01-22T22:20:13Z

comment:48

Does it go away, if you additionally pull in #31228?

This one just disables -march=native regardless, if the compiler of system python does not understand it. What might be more clever, is to just disable -march=native in the definition of sdh_pip_install.

mkoeppe · 2021-01-22T22:42:31Z

comment:49

Replying to @zlscherr:

Is this supposed to fix issues around using /usr/bin/python3 on Big Sur?

No, it's only a first step

zlscherr · 2021-01-22T23:55:48Z

comment:50

Replying to @kliem:

Does it go away, if you additionally pull in #31228?

This one just disables -march=native regardless, if the compiler of system python does not understand it. What might be more clever, is to just disable -march=native in the definition of sdh_pip_install.

Thanks! Yes, with this ticket and #31228 I can get python packages (at least cython) to build again using system python. I'll continue to check to make sure everything builds.

kliem · 2021-01-23T10:07:11Z

comment:51

I think this is good to go.

kliem · 2021-01-23T10:07:11Z

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/502363182, Jonathan Kliem to Jonathan Kliem

mkoeppe · 2021-01-23T17:41:02Z

Changed author from Matthias Koeppe to Matthias Koeppe, Jonathan Kliem

mkoeppe · 2021-01-23T17:41:02Z

Changed reviewer from Jonathan Kliem to Jonathan Kliem, Matthias Koeppe

vbraun · 2021-01-31T20:53:57Z

Changed branch from u/gh-kliem/macos__spkg_install__remove_outdated__building_with_clang__code___which_conflicts_with___march_native_ to f56e65e

mkoeppe added this to the sage-9.3 milestone Oct 5, 2020

mkoeppe added c: porting labels Oct 5, 2020

mkoeppe changed the title ~~macOS: spkg-install: Remove outdated "Building with clang" branches~~ macOS: spkg-install: Remove outdated "Building with clang" code - which conflicts with '-march=native' Jan 16, 2021

mkoeppe added p: blocker / 1 and removed p: minor / 4 labels Jan 16, 2021

This comment has been minimized.

Sign in to view

mkoeppe added the s: needs review label Jan 18, 2021

mkoeppe added s: needs work and removed s: needs review labels Jan 18, 2021

This comment has been minimized.

Sign in to view

kliem added s: positive review and removed s: needs review labels Jan 23, 2021

vbraun removed the s: positive review label Jan 31, 2021

vbraun closed this as completed in f0ca7df Jan 31, 2021

macOS: spkg-install scripts that force use of clang conflict with '-march=native' #30725

macOS: spkg-install scripts that force use of clang conflict with '-march=native' #30725

Comments

mkoeppe commented Oct 5, 2020

This comment has been minimized.

mkoeppe commented Jan 16, 2021

mkoeppe commented Jan 18, 2021

This comment has been minimized.

mkoeppe commented Jan 18, 2021

kiwifb commented Jan 18, 2021

kiwifb commented Jan 18, 2021

This comment has been minimized.

mkoeppe commented Jan 18, 2021

This comment has been minimized.

sagetrac-git mannequin commented Jan 18, 2021

sagetrac-git mannequin commented Jan 18, 2021

mkoeppe commented Jan 18, 2021

kliem commented Jan 18, 2021

mkoeppe commented Jan 18, 2021

kliem commented Jan 18, 2021

mkoeppe commented Jan 18, 2021

mkoeppe commented Jan 18, 2021

dimpase commented Jan 18, 2021

mkoeppe commented Jan 18, 2021

This comment has been minimized.

mkoeppe commented Jan 19, 2021

kliem commented Jan 20, 2021

kliem commented Jan 20, 2021

kliem commented Jan 20, 2021

kliem commented Jan 20, 2021

kliem commented Jan 20, 2021

mkoeppe commented Jan 20, 2021

kliem commented Jan 20, 2021

mkoeppe commented Jan 20, 2021

kliem commented Jan 20, 2021

kliem commented Jan 21, 2021

mkoeppe commented Jan 21, 2021

mkoeppe commented Jan 21, 2021

mkoeppe commented Jan 21, 2021

kliem commented Jan 22, 2021

kliem commented Jan 22, 2021

zlscherr commented Jan 22, 2021

kliem commented Jan 22, 2021

mkoeppe commented Jan 22, 2021

zlscherr commented Jan 22, 2021

kliem commented Jan 23, 2021

kliem commented Jan 23, 2021

mkoeppe commented Jan 23, 2021

mkoeppe commented Jan 23, 2021

vbraun commented Jan 31, 2021