Add stack build --prefix option

[borsboom]

This new stack build option will add a post-build step that rebuilds the targets with a cabal configure --prefix= option in a separate dist directory. This will allow installing packages that use data-files in a fixed location on the file system outside of any stack-controlled sandbox.

[borsboom]

In order to support installation into directories where the user doesn't have write permission (e.g., /usr/local), should support a flag to specify a command to prefix the Setup.hs copy step with (e.g., sudo). Before starting, check whether writing to the destination works, and if it does not, display a friendly error with a suggestion to use the prefixing option.

[borsboom]

Make sure this covers the case outlined in #1262 (comment)

[mgsloan]

Seems like this issue is cropping up for multiple people. Bumping to P2

[borsboom]

Important use case for supporting a --prefix option: with it, it would be feasible for Homebrew to build their packages using Stack (for those packages that have a stack.yaml), which would aid reproducibility. See Homebrew/homebrew-core#1480 and Homebrew/legacy-homebrew#49158 for related discussion.

[borsboom]

Here's a more concrete example for Homebrew: Homebrew/homebrew-core#1630

[Blaisorblade]

Further rationale—sorry if already mentioned in some links: clearing snapshots (to recover space) leaves compiled binaries around, but not their data-files.

[borsboom]

Many packages have data-files that aren't actually required for the package to work (like READMEs and Changelogs), and since this feature would require rebuilding those packages and all packages that depend on them (which could be a lot of packages), it should have a way to skip some or all dependencies.

[mgsloan]

More demand for this feature, bumping priority.

[essandess]

@borsboom @snoyberg @mgsloan @Blaisorblade
At MacPorts we’ve automated stack builds and this has been successful for packages that don’t use datadir.

However, stack’s and cabal’s lack of a GNU-standard capability for DESTDIR and PREFIX to produce relocatable binaries and installs is causing major problems for package managers that need to build with Haskell development tools. E.g. see:

• ihaskell: Submission of a Haskell kernel for Jupyter macports/macports-ports#4706
• Binaries using hlint have hlint_datadir hardcoded to build directory, not destroot ndmitchell/hlint#699
• [WIP] Fix broken Haskell world macports/macports-ports#5050 (comment)
• adblock2privoxy: Bugfix destroot issue, allow config initialization macports/macports-ports#5180

There’s obviously a lot of interest in the capability and the lack of it is holding back deployment of Haskell-based tools.

Are there any plans to tackle this soon?

Related:

• Support relocatable packages haskell/cabal#462
• [nix-local-build] Support --prefix, --datadir, etc haskell/cabal#3586
• Add stack build --prefix option #848
• Is it expected that data files do not get installed to where bin is? #4857

[essandess]

@borsboom

@essandess Oh one other thing: since you're talking about an intermediate staging area, note that executables will have the staged location of the data files hardcoded, which may not be what you want (it looks like you have some symlink workarounds the ihaskell example to compensate for this). Unfortunately this is a limitation to how data files work, but is out of scope for Stack to fix. In fact, the Stack team recommends against using Cabal data files at all, since it makes location portability of executables so difficult (instead, read-only data can be embedded directly into the executable using file-embed).

What about packages that explicitly specify cabal’s data-files option? Take hlint as an example. Heres’s hlint.cabal:
https://github.com/ndmitchell/hlint/blob/ef26ffcbd0425b98bcc5b330b310df9264a31add/hlint.cabal#L17-L26

When stack is used to build this package, what step causes the breakage of the temporary build directory being hardcoded into the binary?

How can this be fixed when stack is used to build such a package?

[essandess]

@jgm
I see that pandoc.cabal has a ton of data-files, https://github.com/jgm/pandoc/blob/0e31483d4358a6d2b4ba96c71237e3f7b32979a1/pandoc.cabal#L42-L179, yet the pandoc binary that stack builds is free of this issue.

Would you please provide any tips or recipes for converting a package to use embedded files and avoid this issue? Is it relatively easy? Would it be reasonable to ask any packages like hlint that have this problem to refactor their code? What’s the recipe to do that? I see https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Data.hs, but I don’t understand how you’ve turned off cabal’s data-files In pandoc.

Hearing your experience about how to fix this issue with code would be much appreciated.

[essandess]

This older post from @ndmitchell appears to provide at least a couple straightforward solutions to cabal‘s hardcoded path problem:

1. Create a Paths_mypackagename.hs file that monkeypatches the default hardcoded cabal path. If this path is relative to the binary, it would be easy to use ../share/mypackage_datadir or such like.
2. One comment mentions that cabal will use the environment variable mypackage_datadir to set the datadir paths encoded in the executable.

Does anyone have any experience or pointers with this cabal behavior? I’d like to start barking up the right tree.

cc: @borsboom @snoyberg @mgsloan @Blaisorblade @jgm @acfoltzer @bubba @phadej @typedrat @23Skidoo @bos @simonmar @christiaanb

[christiaanb]

I did once implement partial support of "relocatable" packages in Cabal: haskell/cabal#2255; it was sufficient to get relocatable Cabal sandboxes: http://qbaylogic.com/blog/2016/05/08/relocatable-sandboxes.html

[essandess]

@christiaanb

I did once implement partial support of "relocatable" packages in Cabal: haskell/cabal#2255; it was sufficient to get relocatable Cabal sandboxes: http://qbaylogic.com/blog/2016/05/08/relocatable-sandboxes.html

Thank you. Does the --enable-relocatable address the problem with hardcoded paths to cabal data-files? If it does, it’s not clear to me.

@christiaanb’s post brings us to three possible solutions:

1. Create a Paths_mypackagename.hs file that monkeypatches the default hardcoded cabal path. If this path is relative to the binary, it would be easy to use ../share/mypackage_datadir or such like.
2. One comment mentions that cabal will use the environment variable mypackage_datadir to set the datadir paths encoded in the executable.
3. cabal --enable-relocatable

@borsboom @snoyberg Would any of these solutions work within stack?

If any of these do work, I could personally modify specific ports of stack-based packages, or even MacPorts automated process for stack builds, but this issue goes above specific packages built on macOS.

If cabal provides this capability, then stack should support it to provide relocatable binaries.

[jgm]

@essandess - pandoc has an embed_data_files cabal flag, which is enabled by default in the stack build. That avoids the issue. This flag causes all the data files to be embedded as bytestring blobs in the binary, making the executable portable. The file-embed package is used for this. See Text.Pandoc.Data.

Of course, this is not the ideal solution when pandoc is installed by a package manager. In that case it's usually good practice to have the data files live separately in the file hierarchy, where they can be inspected and replaced. (This is how debian linux installs of pandoc work, for example.) Unfortunately, stack doesn't currently support this because of the lack of --prefix.

[essandess]

The approach described by hlint's author @ndmitchell solves the issue of hardcoded cabal data-files in the binary.

All that's required is to specify the path within a file called Paths_packagename.hs, which is normally automatically generated by cabal with its own paths. Here is an automated stack build that solves the issue along with related files:

• https://github.com/macports/macports-ports/blob/e18fe88466477b20da595346ebce8eab839b9fbc/www/adblock2privoxy/Portfile
• https://github.com/macports/macports-ports/blob/e18fe88466477b20da595346ebce8eab839b9fbc/www/adblock2privoxy/files/Paths_adblock2privoxy.hs

Also, setting the environment variable packagename_datadir at runtime overrides the binary's hardcoded packagename_datadir path.

In contrast, setting datadir in stack.yaml causes stack to actually try to write to that directory during compilation, which breaks GNU DESDIR capability for package managers. I believe that this inconsistent and breaking behavior is a bug; see #5026.

@borsboom @snoyberg @mgsloan @Blaisorblade @jgm @acfoltzer @bubba @phadej @typedrat @23Skidoo @bos @simonmar @christiaanb
Thank you all for all your longstanding help, comments, and pointers about this issue. I believe that the approach identified by @ndmitchell is sufficient for at least BSD package managers to include stack builds of packages that use cabal's data-files.

[Kritzefitz]

Just wanted to note: AIUI Cabal (the library) actuall does support both DESTDIR and PREFIX. PREFIX is passed via configure's --prefix option while DESTDIR is passed via copys --destdir option.