[nix-local-build] Support --prefix, --datadir, etc

[acfoltzer]

Both new-configure and new-build seem to silently swallow these options at the moment.

[Ericson2314]

Do these flags effect building or just installing? I vaguely recall with --enable-relocatable its all argv[0]-relative, but I'm not sure otherwise.

edit answer seems to be #3473

[ezyang]

They affect building because prefix paths will affect what RPATHs we bake into the libraries we produce, as well as any hardcoded paths via a Paths module.

[phadej]

I tagged this to high priority, as e.g. haddock uses data

[ezyang]

Surely less high priority than getting new-install working, though?

[lukel97]

I would like to take a stab at this if this is still wanted. Are there any pointers as to where to start? I did a bit of digging and found this module choosing some default InstallDirs. Is this a good place to tie it up with ConfigFlags?

cabal/cabal-install/Distribution/Client/ProjectPlanning.hs

Lines 1550 to 1573 in 6a97912

	install_dirs cid
	| shouldBuildInplaceOnly spkg
	-- use the ordinary default install dirs
	= (InstallDirs.absoluteInstallDirs
	pkgid
	(newSimpleUnitId cid)
	(compilerInfo compiler)
	InstallDirs.NoCopyDest
	platform
	defaultInstallDirs) {
	-- absoluteInstallDirs sets these as 'undefined' but we have
	-- to use them as "Setup.hs configure" args
	InstallDirs.libsubdir = "",
	InstallDirs.libexecsubdir = "",
	InstallDirs.datasubdir = ""
	}
	| otherwise
	-- use special simplified install dirs
	= storePackageInstallDirs
	storeDirLayout
	(compilerId compiler)
	cid

[essandess]

I observe that this issue breaks alex and happy builds that require these files in their datadir: /opt/local/share/alex, happy. These are critical path binaries that a cabal build breaks.

Because these tools are necessary to bootstrap and use Haskell world, I’ve thrown in the towel and just used an ugly hack of replacing this incorrect path in the binary with the correct path padded with extraneous /‘s.

port contents alex
Port alex contains:
  /opt/local/bin/alex
  /opt/local/share/alex/AlexTemplate.hs
  /opt/local/share/alex/AlexWrappers.hs

port contents happy
Port happy contains:
  /opt/local/bin/happy
  /opt/local/share/happy/GLR_Base
  /opt/local/share/happy/GLR_Lib
  /opt/local/share/happy/GLR_Lib-ghc
  /opt/local/share/happy/GLR_Lib-ghc-debug
  /opt/local/share/happy/HappyTemplate
  /opt/local/share/happy/HappyTemplate-arrays
  /opt/local/share/happy/HappyTemplate-arrays-coerce
  /opt/local/share/happy/HappyTemplate-arrays-coerce-debug
  /opt/local/share/happy/HappyTemplate-arrays-debug
  /opt/local/share/happy/HappyTemplate-arrays-ghc
  /opt/local/share/happy/HappyTemplate-arrays-ghc-debug
  /opt/local/share/happy/HappyTemplate-coerce
  /opt/local/share/happy/HappyTemplate-ghc

[Mikolaj]

@essandess: That's unfortunate. Does the workaround in #3586 (comment) work for you? Optionally with chroot or whatever other utility lets you fool cabal and all its child processes that it's in another directory that it really is.

[essandess]

@essandess: That's unfortunate. Does the workaround in #3586 (comment) work for you? Optionally with chroot or whatever other utility lets you fool cabal and all its child processes that it's in another directory that it really is.

We’re using store-dir, but I observed this issue whether or not this global option is set.

It’s ancient practice with Unix build tools to use PREFIX and DESTDIR so that the deployed target is independent of the build location, so I admit to being confused why this is so broken in cabal.

The cabal flag --datadir or config option or Paths_NAME.hs provides the path, so just compile that path into the binary. Why should this be a difficult and longstanding issue?

Setting up a chroot jail environment for a package manger install is something that almost certainly will never happen.

[Mikolaj]

I observed this issue whether or not this global option is set.

The $pkg_datadir global option?

I agree this should be just fixed and I don't remember any particular difficulty apart of the sprawling code base and testsuite. Right --- and @fgaz confirms above we don't expect any traps or conflicts, so the "longstanding" is not an indication of technical problems, just social ones.

[essandess]

The $pkg_datadir global option?

How does that one work? I don't see anything in the cabal man page, help string, or User's guide:

$ man cabal | egrep -e pkg[-_]datadir
$ cabal build --help | egrep -e pkg[-_]datadir

https://cabal.readthedocs.io/en/stable/search.html?q=pkg_datadir&check_keywords=yes&area=default

Search Results

Your search did not match any documents. Please make sure that all words are spelled correctly and that you've selected enough categories.

[gbaz]

I'm looking at this now and whew its a complicated mess. Honestly, I'm not sure if datafiles ever really did work at all with v2. @fgaz can you show an example where they work and in particular show where the copying is supposed to take place? It looks to me like neither v2-install nor v2-build actually copies them -- they just bake in a Paths module that says where they would be?

In any case, I think I'm on track to get --datadir only working properly. My approach would be that build itself should copy the files to where the Paths module thinks they should be, otherwise you get a nonworking executable during local development. Likewise install should also copy them. And in both cases they should respect --datadir.

I don't think my brain has the capacity to handle prefix or the other installDir flags in the same PR, nor am I sure how many of them make sense at all.

[fgaz]

First of all: nobody addressed #3586 (comment).

Cabal is like make, cabal-install is like apt. You wouldn't call apt while building a package for arch or fedora, right? The distinction is still there exactly for this reason. Nix, debian, arch... all use Cabal (with appropriate --prefix, --datadir, etc), and use their respective package manager to... do package manager things.
Why does macports have to use cabal-install?

--prefix, --datadir etc should just be removed from v2-* commands. They do nothing, they ended up there by accident: #3549

if using cabal-install is actually needed for distro packaging (though haskell is not at npm-like crazy level of dependencies so I don't see why), try to use it just to generate a build plan to then execute with your distro package manager. haskell.nix does this. Our bootstrap script does something similar. Another example.

If that's still not enough, there's --store dir. Set that to the installation location (not a prefix, more like /opt) and you should be ok, though it's a bit hacky. If that's still not enough, then there are #462 and #3473

---

@gbaz

Honestly, I'm not sure if datafiles ever really did work at all with v2. @fgaz can you show an example where they work and in particular show where the copying is supposed to take place?

Agda uses them, they work.
I can't point to the exact location of the copying right now, but it's when the package gets in the store.

It looks to me like neither v2-install nor v2-build actually copies them -- they just bake in a Paths module that says where they would be?

v2-build sets the paths to the values they would assume if the package was installed in the store. v2-run takes care of overriding that through $pkg_datadir variables. This is why one should not run local executables directly. This is actually pretty nice because it'd make it easier to fix #6919 (no need to change Paths_*.hs and rebuild).
v2-install installs the package in the store as normal, then creates a symlink to the exe. the data files are in the expected paths in the store.

In any case, I think I'm on track to get --datadir only working properly.

I think we should make it clear that this is not like --datadir the low-level Cabal flag (or is it? are you splitting up the store?). Furthermore, data files are not the only problem when copying stuff around. if we want to do this, I think we should focus on a complete, higher level solution like #3473 (maybe erroring out on dynamic builds for now) or #462.

[gbaz]

Agda uses them, they work.
I can't point to the exact location of the copying right now, but it's when the package gets in the store.

Ah I see. Thanks!

cabal/cabal-install/src/Distribution/Client/ProjectBuilding.hs

Line 1007 in ebfd8c7

setup Cabal.copyCommand (copyFlags tmpDirNormalised)

buildAndInstallUnpackedPackage calls cabal copy and cabal copy in fact runs install as per:

cabal/Cabal/src/Distribution/Simple.hs

Line 568 in 0abbe37

copyHook = \desc lbi _ f -> install desc lbi f,

Which in turn is the One True place that datafiles are copied to where they're assigned.

That said, I don't understand the objection to just letting the --datadir flag set the underlying --datadir that eventually gets sent to cabal copy? Sure that means that the installed exe will be in the store (or whereever the storedir was set to) and its datadir will be wherever it was set to be, which is not there. But then at least most of the time we get an executable that is more relocatable than before.

I'm open to the argument that nobody actually needs this. But I also think requiring end users (not distros) to not use cabal install to build binaries that can be shipped around is sort of wild.

So I'm very much of two minds here.

[Mikolaj]

@fgaz: what's your opinion on fixing --datadir for other uses than distro packaging? Is it, in general, useful for packages that take advantage of data files? I can't remember the details right now, but don't some people stick to v1- due to datafiles problems?

@essandess: apologies for the confusion

How does that one work? I don't see anything in the cabal man page, help string, or User's guide

I meant <package_name>_datadir, as in

Cabal/src/Distribution/Simple/Build/PathsModule/Z.hs:    tell "_datadir\")    (\\_ -> getPrefixDirReloc $ "

I thought you were referring to that one. but apparently not. :)

[gbaz]

Ok, I have a funny in-between proposal. cabal install really is for installing into the store, so perhaps we should just remove the --datadir etc flags from cabal install. However, if a user wants to build a package inplace such that they can copy the exe to elsewhere, then allowing the flags is no worse than before, as currently there's nothing useful that gets compiled in anyway.

so I suggest we A) remove the options from cabal install, B) make the options work correctly for cabal build, and C) document this carefully.

This maybe satisfies all concerns?

[mouse07410]

This maybe satisfies all concerns?

No, not quite. It works work fine for statically-linked executables.

The problem is with dynamically-linked executables - breviary l because in addition to copying the executable itself (pretty much anywhere, should run ok regardless), now need to copy all of the shared libraries - and somehow make sure the loader would be able to find them.

[gbaz]

Is this related to support for the existing installDirs flags (--prefix, --datadir, etc)? Or is this another feature you would like? I.e. is it about restoring feature parity with v1-build and the installdirs flags in some other way, or...?