[build system] AutoToolsBuildEnvironment on linux distros with default "lib64" libdir

[weatherhead99]

To help us debug your issue please explain:

• I've read the CONTRIBUTING guide.
• I've specified the Conan version, operating system version and any tool that can be relevant.
• I've explained the steps to reproduce the error or the motivation/use case of the question/suggestion.

On some linux distributions (for example, suse & derivatives), the default libdir for installations is "lib64" on x64 systems, rather than "lib". This prevents nearly every package that uses AutoToolsBuildEnvironment and tools.collect_libs(), because they mostly assume the directory for libraries is "lib".

This could be fixed in many cases, for example, by adding the "--libdir" flag with an appropriate value to "./configure", when we call AutoToolsBuildEnvironment.configure(). I realise this might be controversial, since it might break existing packages that do assume "/lib64" directory. However, my experience in installing conan packages is that essentially none of them currently do this, and I have to patch in the "lib64" directory to the folder= argument in tools.collect_libs on most recipes I use that use autotools.

Note the similar problem doesn't occur in (most) cmake build systems because we do pass -DCMAKE_INSTALL_LIBDIR=lib to the cmake configuration command. So in the spirit of that behaviour, I would see doing it for autotools as well to be simply a matter of consistency.

[lasote]

But if the library directory is lib64, you should add it to the self.cpp_info.libdirs. So you could explicitly call tools.collect_libs(folder="lib64") too. Right?

[weatherhead99]

you are correct, of course, but at present this means modifying just about every conan recipe which uses AutoTools when there isn't a pre-built package for my system. In comparison, even though this otherwise would also happen with the cmake generator, it doesn't, because conan goes to great pains to set up all the relevant cmake variables.

Note: the exact same buildsystem (same compiler, same arch) will produce library directory "lib" on a debian system, but "lib64" on suse.

[lasote]

I see. I don't know exactly what can we do without breaking behavior but we will think about it.

[weatherhead99]

I take the point, but I have so far not found a single recipe that does work.
And note that this would only break behaviour in the case where the recipe has only used tools.collect_libs(folder="lib64"), without also collecting them in "lib", so in fact it only breaks behaviour in the case where the recipe already didn't work on the vast majority of distros in use (debian, ubuntu...). Well actually not strictly true, one could have done something like:

lib_str = "lib64" if distro_uses_lib64() else "lib"

and this would break those recipes. Perhaps it would be better instead to provide more functionality in the linux distro specific tools of conan? I.e. being able to get this information from conan without having to hard code it all the time...

[lasote]

But this is not only a collect_libs issue, right? If the package is built in some distros the lib folder would be different, so the package won't declare correctly where the libraries are located, even without collect_libs.

[weatherhead99]

I think I haven't expressed my intent clearly here: I am asking if we can make the AutotoolsBuildEnvironment pass the flags to autotools such that it outputs libraries into the "lib" dir, even on platforms where it otherwise would use "lib64". Thus, collect_libs wouldn't need to be altered. There is no difference in the built package, the change is happening at package build time.

It's not that pre-built packages would change at all, it would just enable recipes to be build on platforems which use "lib64" without modification of the recipe.

[memsharded]

I am afraid this is not doable without breaking.

I am aware of enterprise users using those distros and they already coded their recipes to account for the "lib64" folder. If conan changes the AutotoolsBuildEnvironment default behavior and start to package to the lib folder instead, their builds will break, because when they go to the lib64 folder in their package() methods, binaries won't be there.

I understand most of the open source conan packages out there might not be prepared for this, and it is natural that you don't find many of breaking cases because those enterprise users are not contributing their recipes as open source.

I wouldn't oppose to some opt-in solution that would make trivially easy to define that behavior, which would benefit to open-source packagers as well.

[weatherhead99]

ok, would it be a good idea to add extra logic into somewhere nearby to "systemPackageTool", as there are similar issues there (in fact i was going to / will open a separate issue about it), namely: "dev extension", on ubuntu/debian etc, development packages are named "-dev", whereas on suse/redhat/fedora they are "-devel". On debian, 32 bit packages are named ":i686" whereas on suse they are "-32bit".

Including some tools like this would really make it easier to actually get a systempackagetool dependency to work across distros. And the "lib" / "lib64" issue is in a similar category. But at the same time, it doesn't quite make sense to actually be in systempckagetool because it isn't related functionality

[Croydon]

This issue should likely be renamed to be more clear. This is a general issue (see #3488) - not limited to AutoToolsBuildEnvironment.

[weatherhead99]

in principle agree, but the thrust of this suggestion was to have all libs installed in "lib" folder, even on platforms where it would by default be "lib64". Changing collect_libs() would solve the problem also, I guess, but it introduces some complications:

what happens if I build the exact same recipe using AutoToolsBuildEnvironment with the same settings, compiler etc on ubuntu, and then on fedora? I will get two packages with the same hash, but where the libraries are in different places. If you've fixed "collect_libs" such that it now finds stuff in lib64 as well, fair enough, but nevertheless the binary output of the same package built on different platforms will be different. Not that we're aiming for fully "reproducible build" level consistency, but consider the following situation, which isn't that contrived, it happens for example when using the protoc binary from a shared build of the protobuf package on linux (and presumably OSX) -

when I call the protoc binary from the package, it can't find libprotoc.so because LD_LIBRARY_PATH isn't set. OK, so now I have to either - build the package from a conanfile.py, which appropriately uses RunEnvironment to make that binary work in subsequent builds, or adapt my own buildsystem (e.g. cmake or autotools) downstream of conan to use that. In the situation in general with the collect_libs solution we now have to always, unconditionally append two things to all those path variables, because we can never know (and have no knowledge from the package, because it isn't a change in settings or options) where the libraries will actually be.

On the other hand, if we can gently herd the packages to be more homogeneous by using "lib" in the AutotoolBuildEnvironment, issues like this don't go away but become less complicated. It is also as I mentioned previously quite consistent with e.g. the cmake build helper, which sets things like CMAKE_OUTPUT_LIBRARY_DIR to "lib", thus the same package built on ubuntu or fedora using a cmake build helper will always put the libraries in lib, regardless of platform. So, a solution based on collect-libs for those packages will in fact erroneously add the "lib64" directory where it isn't needed.

As with everything conan, it's these seemingly small but ultimately really frustrating issues that are teasing out the actual problem of attempting to do a c++ binary package manager with different buildsystems

[lasote]

I will get two packages with the same hash, but where the libraries are in different places. If you've fixed "collect_libs" such that it now finds stuff in lib64 as well, fair enough, but nevertheless the binary output of the same package built on different platforms will be different.

Totally agree, very very weird.

I think the less bad fix is to default the lib folder as @danimtb implemented at #3388. But it is breaking behavior for the users assuming that the artifacts are in lib64. Not adjusting the default lib in Autotools feels to me more like a bug, because is bringing non-deterministic/controllable situations to the conan packages. So I would bet for breaking with an opt-out for keep the old behavior.

[danimtb]

Seems the --libdir parameter is handled under each gcc package for each distribution loading a default spec file that defines those gcc parameters and AutoTools just picks up the default values.

In SUSE: --libdir set to usr/lib64

bash-4.3# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib64/gcc/x86_64-suse-linux/4.8/lto-wrapper
Target: x86_64-suse-linux
Configured with: ../configure --prefix=/usr --infodir=/usr/share/info --mandir=/usr/share/man --libdir=/usr/lib64 --libexecdir=/usr/lib64 --enable-languages=c,c++,objc,fortran,obj-c++,java,ada --enable-checking=release --with-gxx-include-dir=/usr/include/c++/4.8 --enable-ssp --disable-libssp --disable-plugin --with-bugurl=http://bugs.opensuse.org/ --with-pkgversion='SUSE Linux' --disable-libgcj --disable-libmudflap --with-slibdir=/lib64 --with-system-zlib --enable-__cxa_atexit --enable-libstdcxx-allocator=new --disable-libstdcxx-pch --enable-version-specific-runtime-libs --enable-linker-build-id --enable-linux-futex --program-suffix=-4.8 --without-system-libunwind --with-arch-32=i586 --with-tune=generic --build=x86_64-suse-linux --host=x86_64-suse-linux
Thread model: posix
gcc version 4.8.5 (SUSE Linux)

In Ubuntu: --libdir set to usr/lib

conan@b639e99edca5:~$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.2.0-8ubuntu3.2' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3.2)
conan@b639e99edca5:~$

In Fedora: Seems not --libdir defined by default. GCC says default is lib.

[root@29189bcbd007 /]# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --enable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.1.1 20180712 (Red Hat 8.1.1-5) (GCC)

I know those are different versions of GCC, but this gives another reason to make this change in the Autotools build helper as those default specs may change from one version to another.

Another issue might be adjusting --bindir too. Otherwise we might face same issue in the future.