Separate static libraries to dev outputs automatically

[ilya-fedin]

Describe the bug

Looking at space used by nix packages, it seems there's a lot of packages that used Nx space due to having static libraries in out/lib outputs. The first such package I found is llvm.lib used by mesa: it consumes ~300 MB while most of the space is used by static libraries that are helpful only while building something, shared libraries in the package are only around ~100 MB, so it's grown around 3x by that issue. Then, when I started looking at all static libraries in /nix/store with find, rav1e stands out from all other packages. Entire consumed space by rav1e is 89M, librav1e.a is 81M while librav1e.so.0.5.1 is only 3,2M. The package is around 10X bigger than if static libraries were separated and 25X bigger than if there was a lib output with only shared library.

Steps To Reproduce

Steps to reproduce the behavior:

1. nix-store -qR /run/current-system | xargs -I [] find [] -name '*.a' -exec du -h {} \; | sort -h

Expected behavior

Static libraries are in dev output

Additional context

My /nix/store (and nix-store -qR /run/current-system | xargs du -csh | sort -h) is 12GB after nix-collect-garbage -d and I'm trying to understand why it's so big while a similar configuration with other distros would consume a way less space...

Notify maintainers

@lovek323 @7c6f434c @dtzWill @primeos for LLVM, it seems rav1e has no maintainers and I don't know who to mention for a generic solution across all the packages...

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 5.16.11-zen1, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.7.0pre20220221_caf5172`
 - channels(root): `"nixos-22.05pre338578.a2e281f5770"`
 - nixpkgs: `/etc/nixpkgs`

[ilya-fedin]

Summing up all static libraries on my system, it seems they consume 523M

nix-store -qR /run/current-system | xargs -i find {} -name '*.a' | xargs du -ch | tail -1
523M    total

[rnhmjoj]

Haskell packages are another big source of static libraries.
IIRC the haskell libraries are statically linked even if the resulting binaries are not (dynamically linked against C libraries), so this seems another waste of space.

[waldheinz]

For LLVM in particular I recently did this:

1748887

But I agree that a generic solution would be nice, I started hacking on a hook doing just that but patching the cmake target files is not trivial.

[alexshpilkin]

@rnhmjoj Haskell (GHC) libraries can be linked into binaries either dynamically or statically, as desired, it’s just that nearly every library or compiler change amounts to an ABI incompatibility, so the traditional (outside of Nix) benefit of transparent upgrades does not apply, only that of sharing read-only pages. I don’t know which way Nix links by default. (Glibc is always linked dynamically.) Of course, the .a files are as large and as useless at runtime as for any other language that uses them.

[Artturin]

What should be the behaviour of lib.getLib, it's used to sometimes .a files ex:

nixpkgs/pkgs/servers/invidious/lsquic.nix

Lines 85 to 98 in faef939

	"-DBORINGSSL_LIB_crypto=${lib.getLib boringssl'}/lib/libcrypto.a"
	"-DBORINGSSL_LIB_ssl=${lib.getLib boringssl'}/lib/libssl.a"
	"-DZLIB_LIB=${zlib}/lib/libz.so"
	];
	# adapted from lsquic.cr’s Dockerfile
	# (https://github.com/iv-org/lsquic.cr/blob/master/docker/Dockerfile)
	installPhase = ''
	runHook preInstall
	mkdir combinedlib
	cd combinedlib
	ar -x ${lib.getLib boringssl'}/lib/libssl.a
	ar -x ${lib.getLib boringssl'}/lib/libcrypto.a

[ilya-fedin]

replace it by getDev?