Survey the build status of ROOT (curated) on macOS

[yipengsun]

@Svende @manuelfs Let's create an issue to track the build status of this package on various versions of macOS.

BTW, you can also see if the official package works with this command:

nix-shell --pure -p root

Big Sur

The official root package works as nix-shell --pure -p root works.

nix develop fails with this message:

error: builder for '/nix/store/sqym5b5sxva0wg9xjr4ri1fpj5ggiqrv-root-6.24.00.drv' failed with exit code 2;
    last 10 log lines:
    > [ 79%] Linking CXX shared library ../../../lib/libCling.so
    > [ 79%] Built target Cling
    > [ 79%] Built target G__Core
    > Scanning dependencies of target Core
    > [ 79%] Linking CXX shared library ../lib/libCore.so
    > ld: framework not found CoreSymbolication
    > clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
    > make[2]: *** [core/CMakeFiles/Core.dir/build.make:502: lib/libCore.so] Error 1
    > make[1]: *** [CMakeFiles/Makefile2:24441: core/CMakeFiles/Core.dir/all] Error 2
    > make: *** [Makefile:171: all] Error 2
    For full logs, run 'nix log /nix/store/sqym5b5sxva0wg9xjr4ri1fpj5ggiqrv-root-6.24.00.drv'.
error: 1 dependencies of derivation '/nix/store/pplxs4j9b25zzvaccalg1bpq9cplwlq1-root-curated-env.drv' failed to build

Catalina

Works allright.

[manuelfs]

Indeed, installed nix in macOS Catalina 10.15.7 from https://github.com/umd-lhcb/root-curated#install-nix-on-macos

sh <(curl -L https://github.com/numtide/nix-flakes-installer/releases/download/nix-2.4pre20210604_8e6ee1b/install)
sudo vi /etc/nix/nix.conf   ## Insert with "i" and escape with "ESC", save and exit with ":wq"

Added the lines to /etc/nix/nix.conf

experimental-features = nix-command flakes
sandbox = false

and restarted computer.

Then

cd ~/code
git clone git@github.com:umd-lhcb/root-curated
cd root-curated
nix develop

This compiled ROOT 6.24/00 successfully in about 40 minutes.

New nix shells are opened by going to root-curated and running nix develop. I checked that I could compile plot_scripts in 6.24/00 fine.

Now, I need to learn

• How to check which ROOT source code was actually compiled
• How to modify the ROOT source in the nix distribution

[yipengsun]

@manuelfs @Svende Can you list the content in /System/Library/PrivateFrameworks and see if you have the CoreSymbolication folder inside? Maybe Svende hasn't install the command line XCode build tools yet?

[manuelfs]

Here's mine
PrivateFrameworks_Manuel.txt

[Svende]

Here is the output of my CoreSymbolication.framework folder:

.
├── Resources -> Versions/Current/Resources
├── Versions
│   ├── A
│   │   ├── Resources
│   │   │   ├── Info.plist
│   │   │   ├── libCoreSymbolicationLTO.dylib
│   │   │   └── version.plist
│   │   └── _CodeSignature
│   │       └── CodeResources
│   └── Current -> A
└── coresymbolicationd

6 directories, 5 files

[yipengsun]

@manuelfs can you also do tree CoreSymbolication.framework?

[manuelfs]

Ah, certainly. I had to install tree with brew install tree and I get

/System/Library/PrivateFrameworks/CoreSymbolication.framework/
├── CoreSymbolication -> Versions/Current/CoreSymbolication
├── Resources -> Versions/Current/Resources
├── Versions
│   ├── A
│   │   ├── CoreSymbolication
│   │   ├── Resources
│   │   │   ├── Info.plist
│   │   │   ├── libCoreSymbolicationLTO.dylib
│   │   │   └── version.plist
│   │   └── _CodeSignature
│   │       └── CodeResources
│   └── Current -> A
└── coresymbolicationd

6 directories, 7 files

So similar but not quite?

[yipengsun]

Your directory contains one more item: CoreSymbolication, which is a symbolic link. This can have important consequences.

[yipengsun]

This might be related: NixOS/nixpkgs#91748.

[yipengsun]

Ah, certainly. I had to install tree with brew install tree and I get

/System/Library/PrivateFrameworks/CoreSymbolication.framework/
├── CoreSymbolication -> Versions/Current/CoreSymbolication
├── Resources -> Versions/Current/Resources
├── Versions
│   ├── A
│   │   ├── CoreSymbolication
│   │   ├── Resources
│   │   │   ├── Info.plist
│   │   │   ├── libCoreSymbolicationLTO.dylib
│   │   │   └── version.plist
│   │   └── _CodeSignature
│   │       └── CodeResources
│   └── Current -> A
└── coresymbolicationd

6 directories, 7 files

So similar but not quite?

The missing symblink could be due to this: https://gitlab.haskell.org/ghc/ghc/-/issues/18446

[yipengsun]

Also related: NixOS/nixpkgs#98541.

[yipengsun]

It looks like some linker-related issue for Big Sur. I plan to submit a bug report upstream. Before doing that, @Svende @manuelfs can you do one last check:

nix-build --check -A root '<nixpkgs>'

For Svende I think this would lead to the same ld error; for Manuel I think it would either success or complain like this (this is on my computer, with an older nixpkgs version, yours will not be exactly like this):

error: derivation '/nix/store/m5gjxn6yvkwy9barhg08grv8hf4qyjar-root-6.18.04.drv' may not be deterministic: output '/nix/store/cxj084y4vlfhb96gal4lrnyvij1xhn4h-root-6.18.04' differs

But ROOT should still build successfully and you don't need to worry about it.

[manuelfs]

Indeed, it got to 100% and installed all the packages, and at the end it says

/nix/store/nqj258lc43a8jcci2xkfsbjk3a0n5akq-root-6.24.00/etc/dictpch/makepch.py: interpreter directive changed from "#! /usr/bin/env python" to "/nix/store/3dvkfi06mzgszfwkhlrjnww1fj02ymbf-python3-3.8.9/bin/python"
warning: rewriting hashes in '/nix/store/nqj258lc43a8jcci2xkfsbjk3a0n5akq-root-6.24.00'; cross fingers
warning: dumping very large path (> 256 MiB); this may run out of memory
error: derivation '/nix/store/wgzyz9y0cd98d7x0qq63jyb634k7m7iq-root-6.24.00.drv' may not be deterministic: output '/nix/store/8q7xmhx4ak6wx0flylm1phasvh4dh8hd-root-6.24.00' differs

[Svende]

I again have the same problem:

Scanning dependencies of target Cling
[ 79%] Linking CXX shared library ../../../lib/libCling.so
[ 79%] Built target Cling
[ 79%] Built target G__Core
Scanning dependencies of target Core
[ 79%] Linking CXX shared library ../lib/libCore.so
ld: framework not found CoreSymbolication
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [core/CMakeFiles/Core.dir/build.make:502: lib/libCore.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:24441: core/CMakeFiles/Core.dir/all] Error 2
make: *** [Makefile:171: all] Error 2
error: builder for '/nix/store/qvh9x3wys8lvw6lp02agr8qbcppd3bgw-root-6.24.00.drv' failed with exit code 2;
       last 10 log lines:
       > [ 79%] Linking CXX shared library ../../../lib/libCling.so
       > [ 79%] Built target Cling
       > [ 79%] Built target G__Core
       > Scanning dependencies of target Core
       > [ 79%] Linking CXX shared library ../lib/libCore.so
       > ld: framework not found CoreSymbolication
       > clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
       > make[2]: *** [core/CMakeFiles/Core.dir/build.make:502: lib/libCore.so] Error 1
       > make[1]: *** [CMakeFiles/Makefile2:24441: core/CMakeFiles/Core.dir/all] Error 2
       > make: *** [Makefile:171: all] Error 2
       For full logs, run 'nix log /nix/store/qvh9x3wys8lvw6lp02agr8qbcppd3bgw-root-6.24.00.drv'.

[yipengsun]

Yeah that's to be expected. Thanks for that test so that in my bug report I can just say that the official package doesn't build but the already built one works.

[yipengsun]

@Svende can you also run this command: nix-shell -p nix-info --run "nix-info -m" and paste the result? This will list the pointer of the main package "registry" (nixpkgs) and will be useful for debugging.

[Svende]

Here is the output:
nix-shell -p nix-info --run "nix-info -m"
these 2 paths will be fetched (0.01 MiB download, 0.03 MiB unpacked):
/nix/store/8lhb79fg4n70ixlll090wqlnzd2ahbd8-nix-info
/nix/store/s9r8s3zidsjgsb63l05453g0wsjgqn89-DarwinTools-1
copying path '/nix/store/s9r8s3zidsjgsb63l05453g0wsjgqn89-DarwinTools-1' from 'https://cache.nixos.org'...
copying path '/nix/store/8lhb79fg4n70ixlll090wqlnzd2ahbd8-nix-info' from 'https://cache.nixos.org'...

• system: "x86_64-darwin"
• host os: Darwin 20.3.0, macOS 10.16
• multi-user?: yes
• sandbox: no
• version: nix-env (Nix) 2.4pre20210604_8e6ee1b
• channels(root): "nixpkgs-21.11pre295670.189a1368878"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixpkgs

[yipengsun]

Hmm. Apparently either Apple or Nix team decided to call Big Sur macOS 10.16?!

[yipengsun]

I've made changes according to suggestions in NixOS/nixpkgs#128576.

@Svende can you try this in your local root-curated project root:

git pull
nix develop

And see if ROOT now compiles?

[yipengsun]

Sorry @Svende, please hold on, as porting the changes is more difficult than I thought.

[Svende]

With the latest changes nix develop works for me with ROOT 6.24/00

[yipengsun]

I've updated the nixpkgs pointed to the latest. Now ROOT 6 should build w/o problems both on Big Sur and Catalina.

@manuelfs @Svende Could you both pull the latest changes and run nix develop again to verify that ROOT 6.24.02 builds?

I'll close this issue after your verification and report this upstream.

[manuelfs]

I pulled the latest changes in lhcb-ntuples-gen, typed nix develop, and it went into the shell immediately.

I then also pulled the latest changes in root-curated, typed nix develop, and it took 1:12 to compile.

My questions

• Does this procedure pick up the latest version of nixpkgs? I thought that nixpkgs was something that was installed separately to make nix work, but in your post it seems like something whose version is configured in flake.nix.
• lhcb-ntuples-gen depends on root-curated If I clone them separately, and then compile them, do I get two separate compilations of the same code taking space in my hard drive?

[yipengsun]

• nixpkgs is the main registry that contains all official nix packages. The flake contains a pointer to a specific version of nixpkgs. When you install something like ROOT, its dependencies, like gcc/clang is pulled from that specific version of nixpkgs.

  Typically, this operation WILL NOT pick up the latest version of nixpkgs, because we PIN it in our root-curated.

• No. As long as the root-curated pointer in lhcb-ntuples-gen flake is the same as the stand-alone root-curated, both will use the same version ROOT.

  Let me try to elaborate on the pointer thing. Remember that we PIN our nixpkgs explicitly ONLY in root-curated. On the other hand, we PIN root-curated in lhcb-ntuples-gen, so it's possible that the pinned version of root-curated in lhcb-ntuples-gen is different from your stand-alone root-curated. In this case, it is possible that there will be 2 separate versions of ROOT because of different build dependencies (say, the pinned version in lhcb-ntuples-gen uses gcc9, whereas the stand-alone version uses gcc10).

[yipengsun]

In our flake workflow, no pointer is never updated automatically. If you want to update the pointer (called inputs in flakes), refer to this section.

I think the right model is: Treating a flake like a pure function: It tries very hard to not rely on anything external, so if you use the same input, and you don't change the code of flake itself, it will always produce the same output (that's why in lhcb-ntuples-gen, nix develop returns immediately: It still uses the old pointers/inputs).

[yipengsun]

Let me try to explain the whole hierarchy:

• nixpkgs contains all official packages provided by Nix/NixOS project
  • nix is a package in nixpkgs
• When you install nix following my instruction, a system-wide version of nixpkgs is specified in the installer. Denote this as nixpkgs-ver_sys.
• In the root-curated flake, we PIN a version of nixpkgs. Denote this as nixpkgs-ver_root_curated
  • In general, nixpkgs-ver_sys != nixpkgs-ver_root_curated
  • This nixpkgs-ver_root_curated is only effective in the root-curated project. It is completely isolated from your system.
• In lhcb-ntuples-gen flake, we PIN a version of root-curated. We also configure this flake in a way such that we use the nixpkgs pinned in the pinned root-curated flake. Denote this nixpkgs as nixpkgs-ver_ntuples_gen
  • Similarly, nixpkgs-ver_ntuples_gen != nixpkgs-ver_root_curated, unless you manually update root-curated pointer in lhcb-ntuples-gen.
  • Another possibility of nixpkgs-ver_ntuples_gen == nixpkgs-ver_root_curated is that we don't update the nixpkgs pointer in root-curated.

This diagram may be helpful:

[Svende]

I've updated the nixpkgs pointed to the latest. Now ROOT 6 should build w/o problems both on Big Sur and Catalina.

@manuelfs @Svende Could you both pull the latest changes and run nix develop again to verify that ROOT 6.24.02 builds?

I'll close this issue after your verification and report this upstream.

I don't understand from your discussion from where and in which folder I have to pull the latest changes. Can you be more specific @yipengsun ?

[manuelfs]

I think by default, if a package is not mentioned in a github issue, he refers to the repo in which the issue is hosted, so in this case root-curated.

Based on my findings above and Yipeng's clarifications (thanks for the discussion and the diagram Yipeng), it seems like he updated root-curated with the version of nixpkgs that works in Big Sur.

lhcb-ntuples-gen was still pointing to an old version of root-curated, however, so that's why it didn't do the recompilation when I got into it. When he updates lhcb-ntuples-gen to point to this new version of root-curated we will not have to recompile though, because it seems nix is smart and will just use the recompilation that was done when we pull the latest root-curated

[yipengsun]

OK. Both Manuel and Svende confirmed that ROOT 6.24./02 compiles on both Catalina and Big Sur and upstream's fix works as intended.