Build static librdkafka_windows.a with travis

[neptoess]

Add to the MinGW travis worker to have it build a librdkafka static bundle, per confluentinc/confluent-kafka-go#555 (comment)

[neptoess]

@edenhill
This should be ready to go. Let me know if it needs anything else.

[neptoess]

Well, looks like one of the tests seg faulted in the latest Travis run, but the previous commit (1a589f6) ran that same test without issue, and all the latest commit changed was packaging.py

[edenhill]

Great! Is this ready to be merged?

[neptoess]

@edenhill
Ready to merge

[edenhill]

Superb! Great work (and perseverence) on this, @neptoess !

[edenhill]

Hey @neptoess , can you help me figure out why the Travis-CI mingw jobs are silently failing?

I've added some debugging and it seems like the test-runner.exe fails to execute, but I don't see an error anywhere (stderr and stdout seems to be weirdly mixed up).
Can you try to reproduce this issue locally?
https://app.travis-ci.com/github/edenhill/librdkafka/jobs/546274490#L1046

[edbordin]

I had issues with GCC LTO being broken on MinGW/MSYS at some point in the past year - the resulting executables would just segfault. Is there any chance this is being built with -flto or related flags?

edit: found the issue tracking it msys2/MINGW-packages#8074

[edenhill]

Thanks @edbordin, there doesn't seem to be an -flto flag in the make output though:
https://app.travis-ci.com/github/edenhill/librdkafka/jobs/546447298

[neptoess]

Looking into this. I'm thinking this has to do with MSYS2 updating the MinGW compiler to GCC 11
https://packages.msys2.org/package/mingw-w64-x86_64-gcc

This happened 2021/10/21. When I did pacman -Syu on my existing MSYS2 install, it did not upgrade beyond GCC 10.3. Uninstalling and reinstalling MSYS2, and copying the pacman commands from the Travis script left me with 11.2. Will do some poking around, but I'm guessing there are some libraries being linked into our build that were built with GCC 10.

[neptoess]

I did a fresh MSYS2 install, fresh git pull, and built the tstwinb branch locally. All tests passed. Going to focus on Travis now, since I think the code (both library and build scripts) is fine.
test-run.log

[edenhill]

Thanks Bill!
Going forward, is it possible to version pin the toolchain and dependencies on the mingw builder to avoid this in the future?

[neptoess]

Well, because my local just worked (with a completely fresh MSYS2 install), I'm not confident this is a package version issue. I was going to try https://github.com/neptoess/librdkafka/commit/3ddeb537bdb83fd53e84afc3acd37adff7e76583 (I should have added a --no-confirm so it doesn't hang) to see if pacman reports any packages needing updated after the choco upgrade command in the build script, but, due to the Travis pricing changes, I can't do any Travis builds at the moment (I have to email them for the usage based plan).

Would it be possible to manually trigger a build on a previous commit that worked?

[edenhill]

Yeah, Travis costs are a sudden and steep barrier.
I think you can use my credits though if you open a PR and comment out all the other workers from .travis.yaml so you just have the single mingw one.

[neptoess]

Good idea. Watching it now https://app.travis-ci.com/github/edenhill/librdkafka/builds/241104613

[neptoess]

https://app.travis-ci.com/github/edenhill/librdkafka/builds/241106018
Even rebuilding v1.8.2 failed. Going to be tricky to debug, since a fresh install of MSYS2 on Win10 works fine.

[edenhill]

These are the mingw toolchain version diffs between the proper 1.8.2 build and your build:

--- 543777511s  2021-11-03 19:25:32.823602012 +0100
+++ 241106018s  2021-11-03 19:24:25.243884678 +0100
@@ -3,15 +3,15 @@
  mingw-w64-x86_64-bzip2-1.0.8-2-any downloading...
  mingw-w64-x86_64-ca-certificates-20210119-1-any downloading...
  mingw-w64-x86_64-c-ares-1.17.2-1-any downloading...
- mingw-w64-x86_64-cmake-3.21.3-1-any downloading...
- mingw-w64-x86_64-crt-git-9.0.0.6316.acdc7adc9-1-any downloading...
+ mingw-w64-x86_64-cmake-3.21.4-1-any downloading...
+ mingw-w64-x86_64-crt-git-9.0.0.6327.f29c1101f-1-any downloading...
  mingw-w64-x86_64-curl-7.79.1-1-any downloading...
  mingw-w64-x86_64-expat-2.4.1-1-any downloading...
-mingw-w64-x86_64-gcc-10.3.0-8-any downloading...
- mingw-w64-x86_64-gcc-libs-10.3.0-8-any downloading...
+ mingw-w64-x86_64-gcc-11.2.0-1-any downloading...
+ mingw-w64-x86_64-gcc-libs-11.2.0-1-any downloading...
  mingw-w64-x86_64-gettext-0.19.8.1-10-any downloading...
  mingw-w64-x86_64-gmp-6.2.1-2-any downloading...
- mingw-w64-x86_64-headers-git-9.0.0.6316.acdc7adc9-1-any downloading...
+ mingw-w64-x86_64-headers-git-9.0.0.6327.f29c1101f-1-any downloading...
  mingw-w64-x86_64-isl-0.24-1-any downloading...
  mingw-w64-x86_64-jansson-2.14-1-any downloading...
  mingw-w64-x86_64-jemalloc-5.2.1-2-any downloading...
@@ -26,8 +26,8 @@
  mingw-w64-x86_64-libtasn1-4.17.0-1-any downloading...
  mingw-w64-x86_64-libtre-git-r128.6fb7206-2-any downloading...
  mingw-w64-x86_64-libunistring-0.9.10-4-any downloading...
- mingw-w64-x86_64-libuv-1.42.0-2-any downloading...
- mingw-w64-x86_64-libwinpthread-git-9.0.0.6316.acdc7adc9-1-any downloading...
+ mingw-w64-x86_64-libuv-1.42.0-3-any downloading...
+ mingw-w64-x86_64-libwinpthread-git-9.0.0.6327.f29c1101f-1-any downloading...
  mingw-w64-x86_64-libxml2-2.9.12-3-any downloading...
  mingw-w64-x86_64-lz4-1.9.3-1-any downloading...
  mingw-w64-x86_64-make-4.3-1-any downloading...
@@ -40,7 +40,7 @@
  mingw-w64-x86_64-pkgconf-1.8.0-2-any downloading...
  mingw-w64-x86_64-rhash-1.4.2-1-any downloading...
  mingw-w64-x86_64-windows-default-manifest-6.4-3-any downloading...
- mingw-w64-x86_64-winpthreads-git-9.0.0.6316.acdc7adc9-1-any downloading...
+ mingw-w64-x86_64-winpthreads-git-9.0.0.6327.f29c1101f-1-any downloading...
  mingw-w64-x86_64-xz-5.2.5-2-any downloading...
  mingw-w64-x86_64-zlib-1.2.11-9-any downloading...
  mingw-w64-x86_64-zstd-1.5.0-1-any downloading...

[neptoess]

Hmm. These package versions all exactly match what I have locally. Is there any way to pull the test-runner.exe artifact out of Travis? I'd like to try running it locally, to rule out anything funny going on between Travis's bash shell and the new C runtime.

[edenhill]

Not easily, no.

That shell invocation is a bit messed up output-wise, so maybe we're not seeing an error message for that reason.
Can we try to execute test-runner.exe in a later step and separate shell in travis?

[neptoess]

Well, that's a little confusing. Breaking tests out into a separate step worked.
https://app.travis-ci.com/github/edenhill/librdkafka/jobs/546739494

[edenhill]

That's good news!

Your new run-tests.sh script does not have the PATH update of the original script, maybe that's what messed up the original?
I have a vague memory that PATH also affects library loading on Windows, not sure if that is true for mingw though.

[neptoess]

%PATH% does affect library loading on Windows, but I believe %PATH% is only checked if the DLL isn't found in the working directory first. This is why it's so common to see people modify, e.g. games, by making proxy DLLs and copying them to the bin folder.

Also, looks like https://app.travis-ci.com/github/edenhill/librdkafka/builds/241213359 worked too, so we don't even need to use a different stage in Travis. I'm guessing just tearing down and making a new shell is the fix.

[neptoess]

Opened #3607 to get the fix merged into master. Let me know if you want me to take a different approach

[edenhill]

Thanks for so resolutely taking on this issue, @neptoess, much appreciated!

[neptoess]

No problem. Happy to help