TLS issues on ubuntu 20 focal

[grosser]

don't have reproduction steps ... it works on a plain focal, but we have a ruby build that adds jmalloc and then crashes with this during packing, hoping there is some obvious "you need to set xyz" fix 🤞
(bionic works fine for some reason)

#14 0.229  INFO exe: loading location=/lib/aarch64-linux-gnu/librt.so.1
#14 0.229  WARN exe: interpreter could not be found. static or compressed executable?
#14 0.229  WARN exe: requesting dynamic libraries of the executable without the interpreter
#14 0.230 error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block
#14 ERROR: executor failed running [/bin/bash -xc /usr/bin/magicpak -v $(which ruby) /magicpak]: exit code: 1

[coord-e]

Assuming the same cause as jemalloc/jemalloc#937, I was able to reproduce it as follows:
https://gist.github.com/coord-e/6384621a1916b0c36ce3f906edee2d3c

I think my guess is not that far off since changing FROM to ubuntu:bionic passes the build...

[coord-e]

So adding --disable-initial-exec-tls to ./configure parameters of jemalloc could at least solves the problem.

From magicpak's side, I confirmed that the error goes away by loading jemalloc at the startup of the resolver (by -Wl,--no-as-needed -ljemalloc in compilation, but just the modification of DT_NEEDED should be sufficient) and using RTLD_NOLOAD flag for dlopen(3) to obtain the handler in the resolver. Nevertheless, I need to consider a little more whether this will work in any case...

[grosser]

awesome, I'll give that a try 🤞

[grosser]

FYI tried these and they did not work

LDFLAGS="-Wl,--no-as-needed" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-jemalloc --enable-dtrace --disable-install-doc --enable-shared --disable-initial-exec-tls

LDFLAGS="-Wl,--no-as-needed -ljemalloc" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-jemalloc --enable-dtrace --disable-install-doc --enable-shared --disable-initial-exec-tls

[coord-e]

Sorry for the confusing wording, -Wl,--no-as-needed -ljemalloc is about the implementation of magicpak, and --disable-initial-exec-tls should be sufficient to be passed to ./configure.
If it doesn't work, I will need more time to figure it out. I'll try building jemalloc on my side.

[grosser]

trying to build something reproducible now 🤞

[grosser]

FROM ubuntu:focal as ruby

ARG ARCH

ENV DEBIAN_FRONTEND=noninteractive \
    RUBY_VERSION=3.1.2 \
    GEM_HOME=/usr/bundle \
    BUNDLE_PATH=/usr/bundle \
    BUNDLE_BIN=/usr/bundle/bin \
    BUNDLE_SILENCE_ROOT_WARNING=1 \
    BUNDLE_APP_CONFIG=/usr/bundle \
    PATH=/usr/bundle/bin:$PATH

RUN apt-get update && \
      # Compile and install ruby
      apt-get -y install libffi-dev libgdbm-dev libssl-dev libyaml-dev \
      procps zlib1g-dev libjemalloc-dev libgmp-dev libsasl2-dev libreadline-dev \
      build-essential libbz2-dev libncurses-dev libxml2-dev libxslt-dev xz-utils \
      libtool curl systemtap-sdt-dev && \
      curl -sSLo /tmp/ruby.tar.xz \
      https://cache.ruby-lang.org/pub/ruby/${RUBY_VERSION%.*}/ruby-${RUBY_VERSION}.tar.xz && \
      tar -C /tmp -xJf /tmp/ruby.tar.xz && \
      cd /tmp/ruby-${RUBY_VERSION} && \
      LDFLAGS="-Wl,--no-as-needed" ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --with-jemalloc --enable-dtrace --disable-install-doc --enable-shared --disable-initial-exec-tls && \
      sed -i 's/ENABLE_PATH_CHECK 1/ENABLE_PATH_CHECK 0/' file.c && \
      make && \
      make install && \
      cd /tmp && \
      # Cleanup
      rm -rf /tmp/ruby-${RUBY_VERSION} /tmp/ruby.tar.xz && \
      rm -rf /var/lib/apt/lists/* && \
      echo "install: --no-document" >> /etc/gemrc && \
      echo "update: --no-document" >> /etc/gemrc && \
      mkdir -p "$GEM_HOME" "$BUNDLE_BIN" && \
      chmod 777 "$GEM_HOME" "$BUNDLE_BIN" && \
      # ruby 2.6+ does not ship with `irb`
      # pinning reline and irb due to https://github.com/ruby/reline/issues/95
      [ $(echo $RUBY_VERSION | cut -f2 -d.) -lt 6 ] || gem install reline:0.0.7 irb:1.2.5 cgi:0.3.1 && \
      rm -rf /root/.gem


# pack up the environment
# note: without includes http requests fail see https://github.com/coord-e/magicpak/issues/12
ADD https://github.com/coord-e/magicpak/releases/download/v1.3.2/magicpak-${ARCH}-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak
RUN /usr/bin/magicpak --verbose --include "/lib/${ARCH}-linux-gnu/libnss_*" --include "/lib/${ARCH}-linux-gnu/libresolv*" $(which ruby) /magicpak

produces

docker build --progress plain --build-arg ARCH=aarch64 .
#10 0.226 error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block

same result with --platform linux/amd64 --build-arg ARCH=x86_64

[coord-e]

I thought you were building jemalloc from source... when building jemalloc, ./configure with the --disable-initial-exec-tls flag, the error goes away.

https://gist.github.com/coord-e/8e8355bf3de42bcd24ddd5455a8759d3

[coord-e]

Anyway, I've released v1.4.0 with --experimental-noload-resolver flag to resolve this issue without building jemalloc from source. Can you try it like the following?

https://gist.github.com/coord-e/016f2a879167a07ef6fa0700265d4161

...

# pack up the environment
# note: without includes http requests fail see https://github.com/coord-e/magicpak/issues/12
ADD https://github.com/coord-e/magicpak/releases/download/v1.4.0/magicpak-${ARCH}-unknown-linux-musl /usr/bin/magicpak
RUN chmod +x /usr/bin/magicpak
RUN /usr/bin/magicpak --verbose --include "/lib/${ARCH}-linux-gnu/libnss_*" --include "/lib/${ARCH}-linux-gnu/libresolv*" --experimental-noload-resolver $(which ruby) /magicpak

[grosser]

confirmed working with --experimental-noload-resolver

[grosser]

also works on ubuntu jammy 🎉

[grosser]

#19 (comment) has --disable-initial-exec-tls and it still does not work still:

#9 0.231 error: Unable to lookup shared library: /lib/aarch64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block

also --disable-initial-exec-tls disables a performance-optimization so not having to use it is a plus (... unless --experimental-noload-resolver does the same under the hood)

... so all good afaik

maybe rename to remove experimental- and talk about downsides of --disable-initial-exec-tls in the readme