Haskell language server fails to build on aarch64-darwin

[tom-hodgson]

Steps To Reproduce

taking the most recent commit:

nix-shell -p '(import (fetchTarball "https://github.com/nixos/nixpkgs/archive/533ca3db3cc6a1fd0645d9e800fc836a0f31b290.tar.gz") {}).haskell-language-server'

Errors with

Preprocessing executable 'fourmolu' for fourmolu-0.8.2.0..
Building executable 'fourmolu' for fourmolu-0.8.2.0..
[1 of 2] Compiling Paths_fourmolu   ( dist/build/fourmolu/autogen/Paths_fourmolu.hs, dist/build/fourmolu/fourmolu-tmp/Paths_fourmolu.o, dist/build/fourmolu/fourmolu-tmp/Paths_fourmolu.dyn_o )
[2 of 2] Compiling Main             ( app/Main.hs, dist/build/fourmolu/fourmolu-tmp/Main.o, dist/build/fourmolu/fourmolu-tmp/Main.dyn_o )
Linking dist/build/fourmolu/fourmolu ...
/nix/store/sxvqpkwix4hq49qxkzghc5d3s14nhcy8-clang-wrapper-11.1.0/bin/ld: line 256: 77946 Segmentation fault: 11  /nix/store/mqvr6f5jnas9909hxpil06li4fwlnb1n-cctools-binutils-darwin-949.0.1/bin/ld ${extraBefore+"${extraBefore[@]}"} ${params+"${params[@]}"} ${extraAfter+"${extraAfter[@]}"}
clang-11: error: linker command failed with exit code 139 (use -v to see invocation)
`cc' failed in phase `Linker'. (Exit code: 139)
error: builder for '/nix/store/wyw1wckb3vqvqckhj8i0zrkyrzv4nwha-fourmolu-0.8.2.0.drv' failed with exit code 1;
       last 10 log lines:
       > [54 of 55] Compiling Ormolu.Utils.Fixity ( src/Ormolu/Utils/Fixity.hs, dist/build/Ormolu/Utils/Fixity.o, dist/build/Ormolu/Utils/Fixity.dyn_o )
       > [55 of 55] Compiling Ormolu           ( src/Ormolu.hs, dist/build/Ormolu.o, dist/build/Ormolu.dyn_o )
       > Preprocessing executable 'fourmolu' for fourmolu-0.8.2.0..
       > Building executable 'fourmolu' for fourmolu-0.8.2.0..
       > [1 of 2] Compiling Paths_fourmolu   ( dist/build/fourmolu/autogen/Paths_fourmolu.hs, dist/build/fourmolu/fourmolu-tmp/Paths_fourmolu.o, dist/build/fourmolu/fourmolu-tmp/Paths_fourmolu.dyn_o )
       > [2 of 2] Compiling Main             ( app/Main.hs, dist/build/fourmolu/fourmolu-tmp/Main.o, dist/build/fourmolu/fourmolu-tmp/Main.dyn_o )
       > Linking dist/build/fourmolu/fourmolu ...
       > /nix/store/sxvqpkwix4hq49qxkzghc5d3s14nhcy8-clang-wrapper-11.1.0/bin/ld: line 256: 77946 Segmentation fault: 11  /nix/store/mqvr6f5jnas9909hxpil06li4fwlnb1n-cctools-binutils-darwin-949.0.1/bin/ld ${extraBefore+"${extraBefore[@]}"} ${params+"${params[@]}"} ${extraAfter+"${extraAfter[@]}"}
       > clang-11: error: linker command failed with exit code 139 (use -v to see invocation)
       > `cc' failed in phase `Linker'. (Exit code: 139)
       For full logs, run 'nix log /nix/store/wyw1wckb3vqvqckhj8i0zrkyrzv4nwha-fourmolu-0.8.2.0.drv'.
error: 1 dependencies of derivation '/nix/store/i6h6dqngdbx45nzn379z6f28ljmjwp81-hls-fourmolu-plugin-1.1.0.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/jmn4xd7c6wrrrnfmyxr85mqjca8vz6w2-haskell-language-server-1.8.0.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/iyfkzzif48j8hx2xlphbzp4rf8x3hn2r-haskell-language-server-1.8.0.0.drv' failed to build

It looks similiar to #149692 but I think this (haskell language server) will affect a wider audience.

I'm not convinced its fourmolu (or ormolu) specific because both of these commands work fine for me:

nix-shell -p '(import (fetchTarball "https://github.com/nixos/nixpkgs/archive/533ca3db3cc6a1fd0645d9e800fc836a0f31b290.tar.gz") {}).haskellPackages.fourmolu'
nix-shell -p '(import (fetchTarball "https://github.com/nixos/nixpkgs/archive/533ca3db3cc6a1fd0645d9e800fc836a0f31b290.tar.gz") {}).haskellPackages.ormolu'

I've read that it may be possible to remove fourmolu from the list of plugins, but I've not been able to find how to do this.

Notify maintainers

@maralorn

Metadata

 - system: `"aarch64-darwin"`
 - host os: `Darwin 21.5.0, macOS 12.4`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.7.0`
 - channels(tom): `""`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixpkgs`

[maralorn]

this has come up on reddit already, but I will repeat parts of my answer here, becaus reddit. https://www.reddit.com/r/haskell/comments/xpjibu/comment/iq64vkp/

Sorry you are having such a bad time. Sadly I have no clue about darwin or debugging capabilities for it.

For a temporary workaround, I would accept a PR which disables the broken plugin(s) on M1, assuming it’s a package specific problem.

The reddit user mentions it could be an outdated clang version, I have no clue if that assessments has any merits …

[maralorn]

Oh, sorry, I had overlooked the last line of your report.

We can try

haskell-language-server = disableCabalFlag "fourmolu" super.haskell-language-server.override { hls-fourmolu-plugin = null; }

either in a local override of your nixpkgs (be careful to apply it to haskellPackages.haskell-language-server

or in nixpkgs in pkgs/development/haskell-modules/configuration-darwin.nix after the line saying } // lib.optionalAttrs pkgs.stdenv.isAarch64 { # aarch64-darwin }.

[tom-hodgson]

Thanks very much, that gives me enough to get going. I'll have a play, it may take some time, I seem to end up building a lot when doing this sort of thing.

[maralorn]

Thanks very much, that gives me enough to get going. I'll have a play, it may take some time, I seem to end up building a lot when doing this sort of thing.0

that shouldn‘t be the case. maybe you are not using a cached commit? i am not sure what channel to follow with aarch64 darwin, but master is probably a bad idea. in fact having a look at hydra here: https://hydra.nixos.org/jobset/nixpkgs/nixpkgs-unstable-aarch64-darwin#tabs-jobs I see that all plugins are actually successfully being built (search for hls-). otoh hls itself has indeed failed in the step of building ormolu, so here things are getting complicated. Let me quickly describe the situation:

That this can happen at all is probably caused by some override which makes the ormolu hls uses slightly different from the one used in the hls-ormolu-plugin. That‘s a pretty frustrating situation in itself, but there were technical limitations and I had been working on hls for 2 full days at that point.

Anyway why can the one fail while the other doesn’t? Either the build is somehow flaky and the one job had luck while the relevant one didn’t or the override actually causes the linker problem to happen. I am not willing to place any bets, the first explanation maybe is a bit more likely, especially since our darwin builders have been shaky in the last weeks, but it could be something completely different.

[tom-hodgson]

Thanks again @maralorn, disabling the plugins helps. Here's what I did

Originally, my (much simplified) shell.nix looked like this:

let hsl = nixPkgs.haskell-language-server.override { 
    supportedGhcVersions = [ "924" ];
    dynamic = true;
} 
in nixPkgs.mkShell {
   buildInputs = [ hsl ];
}

If I change it to this

let
   disableCabalFlag = nixPkgs.haskell.lib.disableCabalFlag;
   
   hslWithoutPlugins = nixPkgs.haskellPackages.haskell-language-server.override  {
      hls-fourmolu-plugin = null;
      hls-ormolu-plugin = null;
    };
   withoutFourmoluFlag = disableCabalFlag hslWithoutPlugins "fourmolu";
   hsl = disableCabalFlag withoutFourmoluFlag "ormolu";

in nixPkgs.mkShell {
   buildInputs = [ hsl ];
}

the shell loads, if my project ghc version matched the default "90" I'd be all good.

I was using the top level attribute haskell-language-server (not nixPkgs.haskellPackages.haskell-language-server) because I needed to override dynamic to stops template haskell crashing the HSL (mac), and supportedGhcVersions so the HSL is built with the same version as the project. I couldn't work out how to pass these flags to the haskellPackages.haskell-language-server. Similarly I couldn't work out how to pass the plugin overrides to the top level haskell-language-server.

Dynamic is no longer needed, it has recently been made the default, and as I only need a single ghc version I can specify this also. Now my new shell.nix looks like this:

let
   disableCabalFlag = nixPkgs.haskell.lib.disableCabalFlag;
   
   hslWithoutPlugins = nixPkgs.haskell.packages.ghc924.haskell-language-server.override  {
      hls-fourmolu-plugin = null;
      hls-ormolu-plugin = null;
    };
   withoutFourmoluFlag = disableCabalFlag hslWithoutPlugins "fourmolu";
   hsl = disableCabalFlag withoutFourmoluFlag "ormolu";

in nixPkgs.mkShell {
   buildInputs = [ hsl ];
}

The shell loads again, and I think this will probably work for most people. I can't be sure, because it doesn't work for me, I still have different GHC versions, probably because I'm using two different nix pins (and haskell.nix), I'm going to play with that a bit more now.

[tom-hodgson]

I struggled a bit getting a compatible GHC with the one HSL was built with, I could get the versions to match but I was getting 'GHC ABIs don't match!' error.

I'm also using haskell.nix which made it harder because all I should have in my nix file is:

   hsl = haskell.tool ghcVersion "haskell-language-server" "latest";

In the end, I used a global overlay. I added /Users/me/.config/nix/overlays/default.nix with the following contents:

self: super: 
let 
   disableCabalFlag = super.haskell.lib.disableCabalFlag;
   hsl1 = super.haskell-language-server.override  {
      hls-fourmolu-plugin = null;
      hls-ormolu-plugin = null;
    };
   hsl2 = disableCabalFlag hsl1 "fourmolu";
   hsl = disableCabalFlag hsl2 "ormolu";
in
{
 haskell-language-server = hsl;
}

Then updated my nix path

export NIX_PATH=$NIX_PATH:nixpkgs-overlays=/Users/me/.config/nix/overlays

and for the first time I'm running haskell language server natively on my M1. Thanks to everyone involved.

[maralorn]

@tom-hodgson yeah, in general haskell-language-server from nixpkgs will not work for a project built with haskell.nix, be careful with that. You might get away with it if you don’t use TemplateHaskell.

[amuricys]

I also hit this issue (i am the original reddit poster), and your solution does help for HLS @maralorn , thanks! no idea why it's so hard to understand the overriding APIs for different packages, so your tip is quite useful (to use disableCabalFlag).

however the segfault on clang still occurs in other packages, like ormolu's 0.5.0.1 version and Agda, which is why i don't think the solution here is sustainable and might just reflect a more general issue in nixpkgs for M1 mac, and not specifically the fault of broken packages for the HLS plugins. I created another reddit post on the much more general topic of overriding clang (nicely illustrates how I suck at Nix) https://www.reddit.com/r/Nix/comments/y16z7w/how_to_override_clang_in_macos/
as you say, it might not help at all, but seems like it might still be worth trying, if nothing else to help people (me) get past this conceptual hurdle haha

[SamirTalwar]

This should be fixed in #196781. It's not a true fix for the issue, but works around it because it seems to be caused by heavy use of Template Haskell. Ormolu/Fourmolu has a flag to disable TH usage at the expense of runtime performance (though it's basically unnoticeable).

[sternenseemann]

Is this still an issue? Can this issue be closed?

[srid]

Is this still an issue? Can this issue be closed?

How can I help test this? I have a M1 Pro. Currently, HLS is failing to build because of :

❯ nix run nixpkgs#haskell-language-server -- --version
[1/0/25 built, 6/19/48 copied (78.3/353.3 MiB), 6.2/39.2 MiB DL] fetching hls-rename-plugin-1.0.1.0-doc from https:/
error: cycle detected in build of '/nix/store/mn4cj42d5b99i1iky0k6z0iq0m8jwrix-ormolu-0.5.0.1.drv' in the references of output 'bin' from output 'out'

EDIT: of #140774

[sternenseemann]

Exactly, that's #140774 (comment). Then I'll close this for now and we can revisit this as soon as the blocking issue is fixed.