Cannot build C++14 in docker images

[YannickJadoul]

Hello. I'm in the situation where I have a C++14 extension (using the brilliant (pybind11)[https://github.com/pybind/pybind11] library), but the GCC version installed on the docker images are 4.8 and do not support the C++14 standard.

Is there a way I can get around this, and would still be able to produce manylinux1 wheels?

[njsmith]

Unfortunately there aren't any compilers that both support c++14 and can generate manylinux1-compatible (= centos 5 compatible) builds.

However, there is a plan to define manylinux2 and manylinux3 standards based on centos6 and centos7, respectively. That's the most likely path to getting c++14 support. Currently there's no one actively working on this, because we're a volunteer run project and none of the existing contributors have the time right now, but it would be a pretty straightforward thing for someone to help with if you want. There's a rough checklist of what needs to happen in this email.

[YannickJadoul]

Thank you for this quick reply.
Hmm, given that I've been learning quite a lot about all this, over the last few days, I would indeed find it interesting to help (after digging into the internals of the first manylinux).

Meanwhile, while lacking such a newer standard, what would be the best way forward, on the short term? 'Downgrading' my C++14 code to 11, or building on another system (e.g., Travis CI's Ubuntu 12.04 (precise))?

[njsmith]

If you can downgrade your code to c++11, then that'll be the most reliable option, yeah. If you build on another system, then your wheels won't be manylinux1 compatible, and only manylinux1 wheels are allowed on pypi currently. Technically I suppose you could lie and label your wheels as being manylinux1-compatible when they aren't, but (a) that would be naughty, and (b) since pip uses the platform tag to determine which systems are compatible with which wheels, lying to it sets up a situation where some of your users may get broken installs. (Of course if you're not distributing these wheels publicly on pypi, then this matters much less.)

I think the manylinux2/3 to-do list is actually pretty approachable if you want to go that way; most of it should be possible to copy directly from the manylinux1 equivalents.

In any case, good luck!

[rmcgibbo]

If you're really careful, it might be possible to use a newer version of g++ with tricks like http://archive.is/wkKoy#selection-129.0-144.0 to generate manylinux1-compatible code, but it requires a pretty high degree of compiler kung fu.

[mpharrigan]

Can't you (1) install a new gcc in the docker container and then (2) just vendor libstdc++

[YannickJadoul]

@rmcgibbo Thanks, not sure how easy that is (though I'll pay around a little bit, next week), but it's a very nice read on the actual problem!

@mpharrigan The problem is that I can't find a GCC >= 5 for CentOS 5, so are you suggesting to build from source?

[njsmith]

Building a recent toolchain and then vendoring libstdc++ would indeed work AFAIK, though I don't know how easy it is.

[mpharrigan]

are you suggesting to build from source?

Yeah

[YannickJadoul]

Right, I'll give these different options a shot, over the course of the next week, and see what works best in my case.

[squeaky-pl]

I've built multilib GCC 6.3 for Centos 5 for Porable PyPy project: https://github.com/squeaky-pl/centos-devtools/releases

If you manage to convince manylinux docker image to use this compiler instead it might work.

You would need to untar it to your root and then use /opt/devtools-6.3/bin/<whatever>, possibly by tweaking PATH:

wget https://github.com/squeaky-pl/centos-devtools/releases/download/6.3/gcc-6.3.0-binutils-2.27-x86_64.tar.bz2 -O - | tar -C / -xj

[rmcgibbo]

@squeaky-pl: that's impressive. i'm eager to hear if it works for the manylinux case. if so, that would be fantastic.

[YannickJadoul]

Wow, this is brilliant! Unpacking and compiling seems to be working like a charm, on my local docker image.

However, I'm still running into a problem with auditwheel

[root@e46be390efb1 dist]# auditwheel show praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl 

praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl is consistent with
the following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.3.1',
'GCC_3.0'}, libstdc++.so.6 with versions {'CXXABI_1.3.5',
'CXXABI_1.3.9', 'CXXABI_1.3', 'GLIBCXX_3.4', 'GLIBCXX_3.4.18',
'GLIBCXX_3.4.20', 'CXXABI_1.3.3', 'GLIBCXX_3.4.21'}, libm.so.6 with
versions {'GLIBC_2.2.5'}, libpthread.so.0 with versions
{'GLIBC_2.2.5', 'GLIBC_2.3.2'}, libc.so.6 with versions
{'GLIBC_2.2.5'}

This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would to recompile a new wheel from
source on a system with earlier versions of these libraries, such as
CentOS 5.

So, @squeaky-pl, is your GCC 6.3 also using a more recent version of CXXABI than the standard one on the manylinux1 CentOS 5?

[squeaky-pl]

I am not quite sure because I don't work with C++ at all. The compiler was compiled itself on Centos 5 but of course C++ standard library evolved quite a lot since GCC 4.8.

The PEP itself says:

GLIBC <= 2.5 <-- fine
CXXABI <= 3.4.8 <-- fine
GLIBCXX <= 3.4.9 <-- this looks violated
GCC <= 4.2.0 <-- fine

So probably this is libstdc++.so.6 that needs to be shipped somehow because it's tightly coupled with compiler/C++ standard version I guess, and probably the resulting wheel references libstdc++.so.6 from that /opt/devtools-6.3 subdir.

[squeaky-pl]

@YannickJadoul what about feeding -static-libstdc++ to the compiler/linker?

[YannickJadoul]

Hmmm, I don't know, but I can give that a shot tomorrow. But I'm guessing there's a reason this isn't that standard approach, or not? Any experience with what the size increase of the wheel would be?

[njsmith]

I think the issue here is that auditwheel assumes that you want to use the system libstdc++, so when it sees that you're using a newer version it just says "nope, not manylinux!" instead of invoking its vendoring logic. Maybe there should be a way to tell it that no, you really do want to vendor libstdc++ (or similar).

-static-libstdc++ will make your wheel bigger, but maybe not as much as copying the entire libstdc++.so into your wheel, which is the other option. (Assuming we're sticking with manylinux1 here.) I guess if you have just one extension module using c++ then -static-libstdc++ should produce a smaller wheel than vendoring, and if you have multiple extension modules using c++ then it's hard to predict.

[YannickJadoul]

That dóes work!

[root@485583349e21 Parselmouth]# auditwheel show dist/praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl 

praat_parselmouth-0.1.0-cp27-cp27m-linux_x86_64.whl is consistent with
the following platform tag: "manylinux1_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libgcc_s.so.1 with versions {'GCC_3.0',
'GCC_3.3', 'GCC_3.3.1', 'GCC_4.2.0'}, libpthread.so.0 with versions
{'GLIBC_2.3.2', 'GLIBC_2.2.5'}, libc.so.6 with versions {'GLIBC_2.3',
'GLIBC_2.2.5'}, libm.so.6 with versions {'GLIBC_2.2.5'}

The following external shared libraries are required by the wheel:
{
    "libc.so.6": "/lib64/libc-2.5.so",
    "libgcc_s.so.1": "/lib64/libgcc_s-4.1.2-20080825.so.1",
    "libm.so.6": "/lib64/libm-2.5.so",
    "libpthread.so.0": "/lib64/libpthread-2.5.so"
}

Gives me an increase in size from about 2.1 MB (2122551) to 2.5 MB (2510676), which seems rather reasonable for a 2017 library.
On the other hand, it still feels somehow 'wrong' in principle, and like a 'temporary hack', or not?

[njsmith]

Sweet.

It's not really any more of a hack then anything else involved in shipping portable binaries – it's just that this time you got a peek behind the covers at the kind of thing that the docker image and auditwheel are trying to hide from you :-).

In the longer run switching to manylinux2/3 is probably a better solution (if only so you don't have to keep maintaining your own toolchain forever!), but I don't think you need to feel bad about it or anything.

Btw parselmouth looks super cool.

[squeaky-pl]

I would say it would be cool (even in the future manylinux2) to let people use whatever C++ standard version they want especially when C++ goes so fast these days. It would be nice to show people that they can use newer compiler and get more manylinux packages, I don't mind maintaining that GCC because I've been doing that already for 3 years.

[njsmith]

@squeaky-pl: it's much less of an issue though for manylinux2/3, because for centos6/7 redhat is maintaining more up to date devtools compilers.

[njsmith]

...though actually it looks like the latest devtoolset compiler is GCC 5.3, so... maybe I'm wrong.

[YannickJadoul]

@njsmith Thanks for the insight! And I dó like the fact that I've now got binary manylinux wheels :-)

I'd actually still be interested in helping with the manylinux2/3, on the longer term, though I don't think I have all the necessary practical knowledge and experience to do all this.

Also, the main 'magic' of Parselmouth is still in the pybind11 library, as far as I'm concerned; that thing is really awesome ;-)

[YannickJadoul]

@squeaky-pl Just out of curiosity: did you actually have to come up with some tricks in order to get that CentOS 5 distribution of GCC 6, or was is 'just a matter of compiling' ?

[squeaky-pl]

@YannickJadoul There are no tricks as far as I know. There is "standard GCC bootstraping pain" that might look tricky to somebody who never built GCC from source but it's no different than compiling GCC on a recent distribution in the end. https://github.com/squeaky-pl/centos-devtools you can look into build and build_deps if you are curious.

[rmcgibbo]

I think it would be great if we could add some of the information in this thread as instructions for getting modern C++ working in the manylinux1 container to the wiki or readme or something. I think many people would benefit from them.

…

Sent from my iPhone

On Jun 15, 2017, at 2:23 PM, Yannick Jadoul ***@***.***> wrote: @njsmith Thanks for the insight! And I dó like the fact that I've now got binary manylinux wheels :-) I'd actually still be interested in helping with the manylinux2/3, on the longer term, though I don't think I have all the necessary practical knowledge and experience to do all this. Also, the main 'magic' of Parselmouth is still in the pybind11 library, as far as I'm concerned; that thing is really awesome ;-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[rmcgibbo]

Is there any possibility of getting runtime problems if a user imports two packages that both use compiled c++ code that statically links different versions of libstdc++?

…

Sent from my iPhone

On Jun 15, 2017, at 2:23 PM, Yannick Jadoul ***@***.***> wrote: @njsmith Thanks for the insight! And I dó like the fact that I've now got binary manylinux wheels :-) I'd actually still be interested in helping with the manylinux2/3, on the longer term, though I don't think I have all the necessary practical knowledge and experience to do all this. Also, the main 'magic' of Parselmouth is still in the pybind11 library, as far as I'm concerned; that thing is really awesome ;-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[njsmith]

@rmcgibbo: If you have two extension modules that use C++ internally but that only communicate via python interfaces (i.e., they're not directly linking to each other or smuggling function pointers through pycapsules or something), then you should be OK with either static linking or vendoring libstdc++. (Watch out for cython's cimport functionality, which under the covers involves smuggling function pointers through pycapsules; otherwise I think you'll know if you're doing this.)

Also, that's assuming that you're using the standard Python interpreter. If you have Python embedded inside a C++ program then I think statically linking libstdc++ will be ok, but vendoring libstdc++ using auditwheel would not. So like, if you think someone might want to use your extension module from inside blender, then stick to statically linking libstdc++ or use the system libstdc++; don't try to vendor libstdc++.

[njsmith]

Actually that blender case is a pretty subtle gotcha – maybe that's an argument for not supporting libstdc++ vendoring in auditwheel, and recommending that people who need a newer libstdc++ do static linking.

[squeaky-pl]

Side note. I pushed GCC 7.1 release: https://github.com/squeaky-pl/centos-devtools/releases/tag/7.1

[YannickJadoul]

@squeaky-pl I've finally gotten to using your binaries, and they're great! Thanks! Things work out fine on 64-bit.

However, it seems as if that's ónly 64-bit, since my 32-bit builds are failing? Am I doing something wrong, or can't I use these binaries to build on the i686 manylinux docker image?

(EDIT: more specifically, the error seems to be centered around ccache: error: execv of /opt/devtools-7.1/bin/gcc failed: No such file or directory)

[riddell-stan]

Anaconda's download page puts 64-bit first and the (unlabeled) "Download" button downloads 64-bit: https://www.anaconda.com/distribution/#download-section This would suggest doing so can't be causing that many problems for a reasonably diverse category of Python users with modern computers.

[TkTech]

Excellent @matthew-brett, I did not realize the pypi stats were on BigQuery 👍

[zooba]

There are still a decent number of 32-bit only Windows users (based on Microsoft telemetry, not the python.org downloads), and the "default option" ought to work for them. And 32-bit processes work just fine on 64-bit Windows, which means everybody gets a working download.

(That said, the Store package only has 64-bit. I'm waiting for complaints before adding a 32-bit package there, but haven't heard any yet :) )

The 32-bit version has a few advantages, even on a 64-bit OS (mostly lower memory usage), so I wouldn't expect it to go away. Certainly not for any current releases (e.g. if 3.8 was the first version to not provide 32-bit builds then you'll have to wait until all other versions are gone before finding any benefit). But if we get to a point where 32-bit Windows is no longer present then eventually the 32-bit builds will disappear.

[tstenner]

I've rebuilt @TkTech's excellent docker image for GCC 9.1.0, you can find it at dockerhub or just pull tstenner/manylinux:gcc9.1.0.

[anthrotype]

@tstenner @TkTech I wonder if you could update your tstenner/manylinux:gcc9.1.0 docker image so that it supports Python 3.8?
I'm getting this error when I use it with the latest multibuild:

/io/multibuild/common_utils.sh: line 243: pip: command not found

https://travis-ci.org/fonttools/skia-pathops/builds/618707554

[anthrotype]

@tstenner @TkTech sorry if I insist.. Perhaps have you published a Dockerfile for tstenner/manylinux:gcc9.1.0 somewhere, that I can adpat to use the latest manylinux1 with support for python3.8? Thanks in advance

[YannickJadoul]

@anthrotype You should just be able to take @TkTech's dockerfile, and rebuild that. Since the manylinux1 docker images now contain Python 3.8, just rebuilding and getting the newer images as base should work.

[anthrotype]

ah! thanks, it was in this very thread and I missed that :)

[TkTech]

I'm attempting to automate it with Github Actions so in the future new images should become available any time manylinux or gcc have a release.

[erwincoumans]

tstenner/manylinux:gcc9.1.0

I tried this, but the version of gcc is old, is there anything special to enable a new gcc and g++?

sudo docker run -it -v $(pwd):/io tstenner/manylinux:gcc9.1.0
[root@8ed459d15a31 /]# gcc --version
gcc (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[root@8ed459d15a31 /]# which gcc
/opt/rh/devtoolset-2/root/usr/bin/gcc

[mayeut]

This has been possible in manylinux2014 for quite some time now (gcc 10 toolchain). Older images are EOL (and manylinux_2_24 is deprecated) too will close this one.