Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

compiler-rt: build builtins on darwin #186575

Merged
merged 1 commit into from
Oct 5, 2022
Merged

compiler-rt: build builtins on darwin #186575

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions pkgs/development/compilers/llvm/11/compiler-rt/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{ lib, stdenv, llvm_meta, version, fetch, cmake, python3, libllvm, libcxxabi }:
{ lib, stdenv, llvm_meta, version, fetch, cmake, python3, xcbuild, libllvm, libcxxabi }:

let

Expand All @@ -15,7 +15,8 @@ stdenv.mkDerivation {
inherit version;
src = fetch "compiler-rt" "0x1j8ngf1zj63wlnns9vlibafq48qcm72p4jpaxkmkb4qw0grwfy";

nativeBuildInputs = [ cmake python3 libllvm.dev ];
nativeBuildInputs = [ cmake python3 libllvm.dev ]
++ lib.optional stdenv.isDarwin xcbuild.xcrun;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This won't fly. Hydra doesn't do xcbuild and since this is in the stdenv, we wouldn't get any darwin builds whatsoever.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, what prevents hydra from building xcbuild? I tried to make it stdenv-friendly in #185969 by exposing xcrun separately, which is just some text files generated by Nix. If that doesn't work, then I guess we'll have to patch the CMake files.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that builders don't have no Xcode installed (to my knowledge) and that it's highly impure to run external build tools.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, but we don't really use Xcode or the SDKs, just emulate them. For example, our xcrun is this shell script:

nixpkgs/pkgs/development/tools/xcbuild/wrapper.nix

Lines 54 to 97 in ab26522

xcrun = writeShellScriptBin "xcrun" ''
args=( "$@" )
# If an SDK was requested, check that it matches.
for ((i = 0; i < ''${#args[@]}; i++)); do
case "''${args[i]}" in
--sdk | -sdk)
i=$((i + 1))
if [[ "''${args[i]}" != '${xcrunSdkName}' ]]; then
echo >&2 "xcodebuild: error: SDK \"''${args[i]}\" cannot be located."
exit 1
fi
;;
esac
done
while [ $# -gt 0 ]; do
case "$1" in
--sdk | -sdk) shift ;;
--toolchain | -toolchain) shift ;;
--find | -find | -f)
shift
command -v $1 ;;
--log | -log) ;; # noop
--verbose | -verbose) ;; # noop
--no-cache | -no-cache) ;; # noop
--kill-cache | -kill-cache) ;; # noop
--show-sdk-path | -show-sdk-path)
echo ${sdks}/${sdkName}.sdk ;;
--show-sdk-platform-path | -show-sdk-platform-path)
echo ${platforms}/${xcodePlatform}.platform ;;
--show-sdk-version | -show-sdk-version)
echo ${sdkVer} ;;
--show-sdk-build-version | -show-sdk-build-version)
echo ${sdkBuildVersion} ;;
*) break ;;
esac
shift
done
if ! [[ -z "$@" ]]; then
exec "$@"
fi
'';

And it references fake macOS SDK and platform directories that are generated plist files. Its entire closure is:

$ nix-store -qR /nix/store/7142q0hx4a6pxhy01y3iwhdwngv26a36-xcrun
/nix/store/i7ql4rg0667cc3ppp1s0v133cr2414ww-SDKs
/nix/store/kvji81lb8cgbzcj59qv8v5lqwmpz1kss-Platforms
/nix/store/rdsvxa6vz0hj00jl7f7dn2ik14yfl6p9-bash-5.1-p16
/nix/store/7142q0hx4a6pxhy01y3iwhdwngv26a36-xcrun

I believe it should be fine, but will try to verify. I thought I was already on a clean macOS, working on a machine provisioned by Scaleway, but it seems Xcode is already on it. I'll delete and try a rebuild.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed Xcode and Command-Line Tools, cleaned up my Nix store, then did a rebuild, and stdenv builds correctly. (I did find an impurity in my Swift PR, so good thing we tried!)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay then, I was concerned because it's located under the xcbuild umbrella. Since it isn't really related to xcode, wouldn't it be best to give it a more fitting name and put it somewhere else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess it's related in the sense that xcbuild, xcrun, xcode-select, etc all emulate Xcode tooling. It's probably called xcbuild because the core tool its packaging is facebook/xcbuild? And the rest of it is small Nix shims.

But facebook/xcbuild is also unmaintained. It looks like we have several dozen packages using it, though not sure how many use the xcbuild parts, and how many just need the Nix shims. Also not sure how well it is capable of building newer Xcode projects (including Swift stuff).

It might make sense to move the non-xcbuild stuff to a separate top-level attribute, like xcode-shims? Maybe @matthewbauer has a stronger opinion. 🙂


NIX_CFLAGS_COMPILE = [
"-DSCUDO_DEFAULT_OPTIONS=DeleteSizeMismatch=0:DeallocationTypeMismatch=0"
Expand Down Expand Up @@ -59,8 +60,9 @@ stdenv.mkDerivation {
# extra `/`.
./normalize-var.patch
../../common/compiler-rt/libsanitizer-no-cyclades-11.patch
] ++ lib.optional stdenv.hostPlatform.isAarch32 ./armv7l.patch;

../../common/compiler-rt/darwin-plistbuddy-workaround.patch
./armv7l.patch
];

preConfigure = lib.optionalString stdenv.hostPlatform.isDarwin ''
cmakeFlagsArray+=("-DCMAKE_LIPO=$(command -v ${stdenv.cc.targetPrefix}lipo)")
Expand All @@ -75,8 +77,6 @@ stdenv.mkDerivation {
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(X86 i386)' 'set(X86 i386 i486 i586 i686)'
'' + lib.optionalString stdenv.isDarwin ''
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'foreach(arch ''${ARM64})' 'foreach(arch)'
substituteInPlace cmake/config-ix.cmake \
--replace 'set(COMPILER_RT_HAS_TSAN TRUE)' 'set(COMPILER_RT_HAS_TSAN FALSE)'
'' + lib.optionalString (useLLVM) ''
Expand Down
11 changes: 6 additions & 5 deletions pkgs/development/compilers/llvm/12/compiler-rt/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{ lib, stdenv, llvm_meta, version, fetch, cmake, python3, libllvm, libcxxabi }:
{ lib, stdenv, llvm_meta, version, fetch, cmake, python3, xcbuild, libllvm, libcxxabi }:

let

Expand All @@ -14,7 +14,8 @@ stdenv.mkDerivation {
inherit version;
src = fetch "compiler-rt" "1950rg294izdwkaasi7yjrmadc9mzdd5paf0q63jjcq2m3rdbj5l";

nativeBuildInputs = [ cmake python3 libllvm.dev ];
nativeBuildInputs = [ cmake python3 libllvm.dev ]
++ lib.optional stdenv.isDarwin xcbuild.xcrun;
buildInputs = lib.optional stdenv.hostPlatform.isDarwin libcxxabi;

NIX_CFLAGS_COMPILE = [
Expand Down Expand Up @@ -59,7 +60,9 @@ stdenv.mkDerivation {
# ld-wrapper dislikes `-rpath-link //nix/store`, so we normalize away the
# extra `/`.
./normalize-var.patch
] ++ lib.optional stdenv.hostPlatform.isAarch32 ./armv7l.patch;
../../common/compiler-rt/darwin-plistbuddy-workaround.patch
./armv7l.patch
];

# TSAN requires XPC on Darwin, which we have no public/free source files for. We can depend on the Apple frameworks
# to get it, but they're unfree. Since LLVM is rather central to the stdenv, we patch out TSAN support so that Hydra
Expand All @@ -70,8 +73,6 @@ stdenv.mkDerivation {
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(X86 i386)' 'set(X86 i386 i486 i586 i686)'
'' + lib.optionalString stdenv.isDarwin ''
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(ARM64 arm64 arm64e)' 'set(ARM64)'
substituteInPlace cmake/config-ix.cmake \
--replace 'set(COMPILER_RT_HAS_TSAN TRUE)' 'set(COMPILER_RT_HAS_TSAN FALSE)'
'' + lib.optionalString (useLLVM) ''
Expand Down
15 changes: 8 additions & 7 deletions pkgs/development/compilers/llvm/13/compiler-rt/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{ lib, stdenv, llvm_meta, version, src, cmake, python3, libllvm, libcxxabi }:
{ lib, stdenv, llvm_meta, version, src, cmake, python3, xcbuild, libllvm, libcxxabi }:

let

Expand All @@ -16,7 +16,8 @@ stdenv.mkDerivation {
inherit src;
sourceRoot = "source/compiler-rt";

nativeBuildInputs = [ cmake python3 libllvm.dev ];
nativeBuildInputs = [ cmake python3 libllvm.dev ]
++ lib.optional stdenv.isDarwin xcbuild.xcrun;
buildInputs = lib.optional stdenv.hostPlatform.isDarwin libcxxabi;

NIX_CFLAGS_COMPILE = [
Expand Down Expand Up @@ -62,9 +63,11 @@ stdenv.mkDerivation {
# ld-wrapper dislikes `-rpath-link //nix/store`, so we normalize away the
# extra `/`.
./normalize-var.patch
] # Prevent a compilation error on darwin
++ lib.optional stdenv.hostPlatform.isDarwin ./darwin-targetconditionals.patch
++ lib.optional stdenv.hostPlatform.isAarch32 ./armv7l.patch;
# Prevent a compilation error on darwin
./darwin-targetconditionals.patch
../../common/compiler-rt/darwin-plistbuddy-workaround.patch
./armv7l.patch
];

# TSAN requires XPC on Darwin, which we have no public/free source files for. We can depend on the Apple frameworks
# to get it, but they're unfree. Since LLVM is rather central to the stdenv, we patch out TSAN support so that Hydra
Expand All @@ -75,8 +78,6 @@ stdenv.mkDerivation {
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(X86 i386)' 'set(X86 i386 i486 i586 i686)'
'' + lib.optionalString stdenv.isDarwin ''
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(ARM64 arm64 arm64e)' 'set(ARM64)'
substituteInPlace cmake/config-ix.cmake \
--replace 'set(COMPILER_RT_HAS_TSAN TRUE)' 'set(COMPILER_RT_HAS_TSAN FALSE)'
'' + lib.optionalString (useLLVM) ''
Expand Down
15 changes: 8 additions & 7 deletions pkgs/development/compilers/llvm/14/compiler-rt/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ lib, stdenv, llvm_meta, version
, monorepoSrc, runCommand
, cmake, python3, libllvm, libcxxabi
, cmake, python3, xcbuild, libllvm, libcxxabi
}:

let
Expand All @@ -26,7 +26,8 @@ stdenv.mkDerivation {
inherit src;
sourceRoot = "${src.name}/${baseName}";

nativeBuildInputs = [ cmake python3 libllvm.dev ];
nativeBuildInputs = [ cmake python3 libllvm.dev ]
++ lib.optional stdenv.isDarwin xcbuild.xcrun;
buildInputs = lib.optional stdenv.hostPlatform.isDarwin libcxxabi;

NIX_CFLAGS_COMPILE = [
Expand Down Expand Up @@ -72,9 +73,11 @@ stdenv.mkDerivation {
# ld-wrapper dislikes `-rpath-link //nix/store`, so we normalize away the
# extra `/`.
./normalize-var.patch
] # Prevent a compilation error on darwin
++ lib.optional stdenv.hostPlatform.isDarwin ./darwin-targetconditionals.patch
++ lib.optional stdenv.hostPlatform.isAarch32 ./armv7l.patch;
# Prevent a compilation error on darwin
./darwin-targetconditionals.patch
../../common/compiler-rt/darwin-plistbuddy-workaround.patch
./armv7l.patch
];

# TSAN requires XPC on Darwin, which we have no public/free source files for. We can depend on the Apple frameworks
# to get it, but they're unfree. Since LLVM is rather central to the stdenv, we patch out TSAN support so that Hydra
Expand All @@ -85,8 +88,6 @@ stdenv.mkDerivation {
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(X86 i386)' 'set(X86 i386 i486 i586 i686)'
'' + lib.optionalString stdenv.isDarwin ''
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(ARM64 arm64 arm64e)' 'set(ARM64)'
substituteInPlace cmake/config-ix.cmake \
--replace 'set(COMPILER_RT_HAS_TSAN TRUE)' 'set(COMPILER_RT_HAS_TSAN FALSE)'
'' + lib.optionalString (useLLVM) ''
Expand Down
25 changes: 25 additions & 0 deletions pkgs/development/compilers/llvm/common/compiler-rt/darwin-plistbuddy-workaround.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
CMake tries to read a list field from SDKSettings.plist, but the output of
xcbuild PlistBuddy is incompatible with Apple's. (Plus we don't want it in our
dependencies.)

Simply assume ARM64 is supported by the SDK. We already limit the actual archs
we build for by setting DARWIN_osx_BUILTIN_ARCHS explicitely.

--- a/cmake/builtin-config-ix.cmake
+++ b/cmake/builtin-config-ix.cmake
@@ -97,14 +97,7 @@ if(APPLE)
set(DARWIN_osx_BUILTIN_MIN_VER 10.5)
set(DARWIN_osx_BUILTIN_MIN_VER_FLAG
-mmacosx-version-min=${DARWIN_osx_BUILTIN_MIN_VER})
- set(DARWIN_osx_BUILTIN_ALL_POSSIBLE_ARCHS ${X86} ${X86_64})
- # Add support for arm64 macOS if available in SDK.
- foreach(arch ${ARM64})
- sdk_has_arch_support(${DARWIN_osx_SYSROOT} macosx ${arch} MACOS_ARM_SUPPORT)
- if (MACOS_ARM_SUPPORT)
- list(APPEND DARWIN_osx_BUILTIN_ALL_POSSIBLE_ARCHS ${arch})
- endif()
- endforeach(arch)
+ set(DARWIN_osx_BUILTIN_ALL_POSSIBLE_ARCHS ${X86} ${X86_64} ${ARM64})

if(COMPILER_RT_ENABLE_IOS)
list(APPEND DARWIN_EMBEDDED_PLATFORMS ios)
15 changes: 8 additions & 7 deletions pkgs/development/compilers/llvm/git/compiler-rt/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{ lib, stdenv, llvm_meta, version
, monorepoSrc, runCommand
, cmake, python3, libllvm, libcxxabi
, cmake, python3, xcbuild, libllvm, libcxxabi
}:

let
Expand All @@ -26,7 +26,8 @@ stdenv.mkDerivation {
inherit src;
sourceRoot = "${src.name}/${baseName}";

nativeBuildInputs = [ cmake python3 libllvm.dev ];
nativeBuildInputs = [ cmake python3 libllvm.dev ]
++ lib.optional stdenv.isDarwin xcbuild.xcrun;
buildInputs = lib.optional stdenv.hostPlatform.isDarwin libcxxabi;

NIX_CFLAGS_COMPILE = [
Expand Down Expand Up @@ -72,9 +73,11 @@ stdenv.mkDerivation {
# ld-wrapper dislikes `-rpath-link //nix/store`, so we normalize away the
# extra `/`.
./normalize-var.patch
] # Prevent a compilation error on darwin
++ lib.optional stdenv.hostPlatform.isDarwin ./darwin-targetconditionals.patch
++ lib.optional stdenv.hostPlatform.isAarch32 ./armv7l.patch;
# Prevent a compilation error on darwin
./darwin-targetconditionals.patch
../../common/compiler-rt/darwin-plistbuddy-workaround.patch
./armv7l.patch
];

# TSAN requires XPC on Darwin, which we have no public/free source files for. We can depend on the Apple frameworks
# to get it, but they're unfree. Since LLVM is rather central to the stdenv, we patch out TSAN support so that Hydra
Expand All @@ -85,8 +88,6 @@ stdenv.mkDerivation {
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(X86 i386)' 'set(X86 i386 i486 i586 i686)'
'' + lib.optionalString stdenv.isDarwin ''
substituteInPlace cmake/builtin-config-ix.cmake \
--replace 'set(ARM64 arm64 arm64e)' 'set(ARM64)'
substituteInPlace cmake/config-ix.cmake \
--replace 'set(COMPILER_RT_HAS_TSAN TRUE)' 'set(COMPILER_RT_HAS_TSAN FALSE)'
'' + lib.optionalString (useLLVM) ''
Expand Down