tox.ini: Add environments for testing with llvm / clang #30835

mkoeppe · 2020-10-27T16:05:27Z

https://groups.google.com/d/msg/sage-devel/5lzj9n57oRI/09AtJyUBBQAJ

We add

a tox configuration factor for using llvm
a script package (without spkg-install) representing system package llvm (https://repology.org/project/llvm/versions)

Examples:

tox -e local-homebrew-macos-standard-llvm
tox -e local-homebrew-macos-standard-usrlocal-llvm (use at your own risk - updates the global homebrew installation)
tox -e docker-fedora-34-standard-llvm

Portability issues (solving them should not be a prerequisite for the present ticket):

When it works on at least a few platforms, we will want to create GH Actions workflows that build Sage with llvm on various platforms for each release tag.

CC: @dimpase @jhpalmieri @videlec @kiwifb

Component: porting

Author: Matthias Koeppe

Branch/Commit: 38e3d3e

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/30835

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-02-13T20:51:01Z

comment:1

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

mkoeppe · 2021-07-09T22:56:56Z

comment:3

Is there a need to distinguish several versions of clang / LLVM on the same system for portability testing purposes?

dimpase · 2021-07-10T10:35:47Z

comment:4

Replying to @mkoeppe:

Is there a need to distinguish several versions of clang / LLVM on the same system for portability testing purposes?

that's probably too much trouble. There is a distinction w.r.t. which standard C++ library clang uses, GNU's (libstdc++) or LLVM's (libc++), which is more important.

mkoeppe · 2021-07-10T17:52:21Z

Branch: u/mkoeppe/tox_ini__gh_actions__add_environments_for_testing_with_llvm___clang

mkoeppe · 2021-07-10T17:54:17Z

Commit: 3200b93

mkoeppe · 2021-07-10T17:54:17Z

comment:6

Here's some preliminary distro package information - just guesswork based on reading https://repology.org/project/llvm/versions

Help with refining it is welcome

New commits:

`3200b93`	`build/pkgs/llvm: New`

mkoeppe · 2021-07-10T18:03:05Z

comment:7

What's the Fortran situation with LLVM? Use flang or gfortran?

sagetrac-git · 2021-07-10T18:17:41Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8b16cf8`	`tox.ini, build/bin/write-dockerfile.sh: Add configuration factor 'llvm'`

sagetrac-git · 2021-07-10T18:17:41Z

Changed commit from 3200b93 to 8b16cf8

sagetrac-git · 2021-07-10T19:11:20Z

Changed commit from 8b16cf8 to 38e3d3e

sagetrac-git · 2021-07-10T19:11:20Z

Branch pushed to git repo; I updated commit sha1. New commits:

`38e3d3e`	`tox.ini [homebrew-llvm]: Set CC, CXX to full path`

dimpase · 2021-07-10T19:41:42Z

comment:11

Replying to @mkoeppe:

What's the Fortran situation with LLVM? Use flang or gfortran?

flang is still not quite ready for prime time
(I managed to build Sage using flang/clang, but it was not stable)

gfortran is good.

mkoeppe · 2021-07-10T19:55:12Z

Author: Matthias Koeppe

dimpase · 2021-07-13T10:42:18Z

comment:15

I tried tox -e local-direct-llvm on Fedora 32, seems to work.

dimpase · 2021-07-13T10:47:39Z

comment:16

How do I actually run tests in such a local setup (I can't use docker on that machine)

mkoeppe · 2021-07-13T16:27:05Z

comment:17

You can do tox -e local-direct-llvm -- ptest, or tox -e local-direct-llvm -- bash if you want a shell

jhpalmieri · 2021-07-13T16:58:41Z

comment:18

I feel like I've had this happen before: I ran tox -e local-homebrew-macos-standard-llvm and by yesterday at 18:13 (according to the timestamp on the file in logs/pkgs) it said " [networkx-2.5.1] successfully installed." Then it sat there with no apparent activity until I hit ctrl-C today at 09:56.

mkoeppe · 2021-07-13T17:04:34Z

comment:19

Prime suspect is gfan's testsuite - see #32088

jhpalmieri · 2021-07-13T20:24:01Z

comment:21

I reran and gfan's testsuite passed. (I had deleted the old log files, so I don't know if that was the problem the first time.) pillow failed to build, though. (This is OS X 11.4.)

jhpalmieri · 2021-07-13T20:24:15Z

Attachment: pillow-8.1.2.log

mkoeppe · 2021-07-13T22:06:03Z

comment:22

Replying to @jhpalmieri:

pillow failed to build, though. (This is OS X 11.4.)

I think this is just that some of Apple's system headers can only be compiled with Apple's compiler.

jhpalmieri · 2021-07-13T23:29:17Z

comment:23

I asked a question like this before, but should there be a make target (or similar) that cleans up the .tox directory? Or if we are using tox, are we just supposed to know what we're doing? Maybe this is relevant.

mkoeppe · 2021-07-13T23:49:29Z

comment:24

I think what you want, in addition to just removing the .tox directory, is #31111 (tox -e local-direct: Clean up the effects of previous tox -e local-....)

dimpase · 2021-07-15T10:09:29Z

comment:25

Replying to @mkoeppe:

Replying to @jhpalmieri:

pillow failed to build, though. (This is OS X 11.4.)

I think this is just that some of Apple's system headers can only be compiled with Apple's compiler.

No, if you just do

+++ b/build/pkgs/pillow/spkg-install.in
@@ -10,7 +10,7 @@ if [ "$UNAME" = "Darwin" ] ; then
     # #29019
     # https://github.com/python-pillow/Pillow/issues/3438#issuecomment-555019284
     # https://apple.stackexchange.com/questions/372032/usr-include-missing-on-macos-catalina-with-xcode-11/372600#372600
-    export CPATH="$CPATH:`xcrun --show-sdk-path`/usr/include"
+    export CFLAGS="$CFLAGS -DTARGET_OS_IPHONE=0 -DTARGET_OS_OSX=1 -DTARGET_OS_MACCATALYST=0"
 fi
 
 if [ "$CONDA_PREFIX" != "" ]; then

then Pillow builds with Homebrew's clang (perhaps there is a more intelligent way, including some header?).

dimpase · 2021-07-15T11:51:38Z

comment:26

with the patch in comment:25, I can build and doctest Sage on Homebrew llvm/clang/clang++ just fine (modulo few known errors).
(perpending /usr/local/opt/llvm/bin to PATH)

mkoeppe · 2021-07-15T16:44:38Z

comment:27

Upstream report?

dimpase · 2021-07-15T17:40:36Z

comment:28

Replying to @mkoeppe:

Upstream report?

do you have any idea what Pillow is doing in a non-standard way while building Python extensions?
(after all, everything else builds, and these TARGET_OS_... are used in the SDK's stdio.h and stdlib.h)

Perhaps it's a setuptools bug?

mkoeppe · 2021-07-15T17:55:26Z

comment:29

Pillow's setup.py does a lot of "discovery", the details are messy. To isolate what is happening here, you could use the PILLOW_BUILD_EXT setting from tox.ini that I use for the macos-nohomebrew environment

mkoeppe · 2021-07-15T17:58:57Z

Replying to @mkoeppe:

Portability issues (solving them should not be a prerequisite for the present ticket):

Let me emphasize that the scope of this ticket should be to add the test environments, which will facilitate testing and porting.

It is not within the scope to do all the porting work.

dimpase · 2021-07-15T18:32:29Z

comment:31

Replying to @mkoeppe:

Pillow's setup.py does a lot of "discovery", the details are messy. To isolate what is happening here, you could use the PILLOW_BUILD_EXT setting from tox.ini that I use for the macos-nohomebrew environment

I am not sure I understand how to
"use the PILLOW_BUILD_EXT setting from tox.ini"

dimpase · 2021-07-15T19:57:16Z

comment:32

If I

export PILLOW_BUILD_EXT="--disable-platform-guessing --disable-jpeg2000 --disable-imagequant --disable-tiff --disable-lcms --disable-webp --disable-webpmux --disable-xcb"

it cannot find zlib. If I remove --disable-platform-guessing it fails with

building 'PIL._imagingmath' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -O2 -g -march=native -g -I/usr/local/Cellar/jpeg/9d/include -I/usr
/local/Cellar/openjpeg/2.4.0/include/openjpeg-2.4 -I/usr/local/Cellar/libtiff/4.3.0/include -I/Users/dima/sage/local/var/tmp/sage/build/pillow-8.1.2/src -I/usr/local/Cellar/freetype/2.10.4/include/freetype2 -I/usr/local/Cellar/little-cms2
/2.12/include -I/Users/dima/sage/local/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/usr/local/include -I/usr/local/Cellar/freetype/2.10.4/include -I/Users/dima/sag
e/local/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/_imagingmath.c -o build/temp.macosx-11-x86_64-3.9/src/_imagingmath.o
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -O2 -g -march=native -g -DHAVE_LIBZ -DPILLOW_VERSION=\"8.1.2\" -I/
usr/local/Cellar/jpeg/9d/include -I/usr/local/Cellar/openjpeg/2.4.0/include/openjpeg-2.4 -I/usr/local/Cellar/libtiff/4.3.0/include -I/Users/dima/sage/local/var/tmp/sage/build/pillow-8.1.2/src -I/usr/local/Cellar/freetype/2.10.4/include/fr
eetype2 -I/usr/local/Cellar/little-cms2/2.12/include -I/Users/dima/sage/local/include -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/usr/local/include -I/usr/local/Cellar/fr
eetype/2.10.4/include -I/Users/dima/sage/local/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/_imaging.c -o build/temp.macosx-11-x86_64-3.9/src/_imaging.o
In file included from In file included from src/_imagingft.c:22:
In file included from /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:64:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/_stdio.h:93:16: warning: pointer is missing a nullability type specifier (_Nonnull, _Nullable, or _Null_unspecified) [-Wnullability
-completeness]
        unsigned char   *_base;
                        ^
...
In file included from /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdio.h:220:5: error: 'TARGET_OS_IPHONE' is not defined, evaluates to 0 [-Werror,-Wundef-prefix=TARGET_OS_]
#if TARGET_OS_IPHONE
    ^
...
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/stdlib.h:181:5: error: 'TARGET_OS_IPHONE' is not defined, evaluates to 0 [-Werror,-Wundef-prefix=TARGET_OS_]
#if TARGET_OS_IPHONE
   ^

dimpase · 2021-07-16T10:30:22Z

Reviewer: Dima Pasechnik

dimpase · 2021-07-16T10:30:22Z

comment:33

OK, let's deal with Homebrew clang on #32207

mkoeppe · 2021-07-16T14:31:51Z

comment:34

Thanks

vbraun · 2021-07-23T20:11:31Z

Changed branch from u/mkoeppe/tox_ini__gh_actions__add_environments_for_testing_with_llvm___clang to 38e3d3e

mkoeppe added this to the sage-9.3 milestone Oct 27, 2020

mkoeppe added c: porting labels Oct 27, 2020

mkoeppe modified the milestones: sage-9.3, sage-9.4 Feb 13, 2021

mkoeppe changed the title ~~tox.ini: Test with homebrew llvm~~ tox.ini, GH Actions: Add environments for testing with llvm / clang Jul 9, 2021

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~tox.ini, GH Actions: Add environments for testing with llvm / clang~~ tox.ini: Add environments for testing with llvm / clang Jul 10, 2021

mkoeppe added the s: needs review label Jul 10, 2021

dimpase added s: positive review and removed s: needs review labels Jul 16, 2021

vbraun removed the s: positive review label Jul 23, 2021

vbraun closed this as completed in 42577e8 Jul 23, 2021

dimpase mentioned this issue Mar 27, 2022

homebrew llvm / clang - followup to #30835 #32207

Closed

tox.ini: Add environments for testing with llvm / clang #30835

tox.ini: Add environments for testing with llvm / clang #30835

Comments

mkoeppe commented Oct 27, 2020

mkoeppe commented Feb 13, 2021

mkoeppe commented Jul 9, 2021

This comment has been minimized.

dimpase commented Jul 10, 2021

mkoeppe commented Jul 10, 2021

mkoeppe commented Jul 10, 2021

mkoeppe commented Jul 10, 2021

mkoeppe commented Jul 10, 2021

sagetrac-git mannequin commented Jul 10, 2021

sagetrac-git mannequin commented Jul 10, 2021

sagetrac-git mannequin commented Jul 10, 2021

sagetrac-git mannequin commented Jul 10, 2021

This comment has been minimized.

dimpase commented Jul 10, 2021

mkoeppe commented Jul 10, 2021

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

dimpase commented Jul 13, 2021

dimpase commented Jul 13, 2021

mkoeppe commented Jul 13, 2021

jhpalmieri commented Jul 13, 2021

mkoeppe commented Jul 13, 2021

jhpalmieri commented Jul 13, 2021

jhpalmieri commented Jul 13, 2021

mkoeppe commented Jul 13, 2021

jhpalmieri commented Jul 13, 2021

mkoeppe commented Jul 13, 2021

dimpase commented Jul 15, 2021

dimpase commented Jul 15, 2021

mkoeppe commented Jul 15, 2021

dimpase commented Jul 15, 2021

mkoeppe commented Jul 15, 2021

mkoeppe commented Jul 15, 2021

dimpase commented Jul 15, 2021

dimpase commented Jul 15, 2021

dimpase commented Jul 16, 2021

dimpase commented Jul 16, 2021

mkoeppe commented Jul 16, 2021

vbraun commented Jul 23, 2021