Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setuptools-rust fails to cross-compile cryptography using crossenv #294

Closed
th0ma7 opened this issue Sep 28, 2022 · 10 comments
Closed

setuptools-rust fails to cross-compile cryptography using crossenv #294

th0ma7 opened this issue Sep 28, 2022 · 10 comments

Comments

@th0ma7
Copy link

th0ma7 commented Sep 28, 2022

I'm looking at getting guidance in order to get to building any rust python wheel properly under our framework under the SynoCommunity spksrc project, starting with cryptography.

We're using crossenv with all the necessary dependency bits AFAICT and we're building all wheels using pip either as cross-compiled or pure-python as needed.

Note that while learning on the topic I ended-up figuring that --target wasn't the way to go as it ends-up building rustc host plugins (crates?) using the cross-compiling environment making things worst as they end-up being built for the target and impossible to run on the host.

As such I'm now using env variables such as CARGO_BUILD_TARGET="armv7-unknown-linux-gnueabihf" (platform dependent) which seems to bring me closer to my goal.

The issue I believe I've been hitting is that, using pip the build environment, does not seems to find the proper linker. By default the cross-compiling environment sets all the CC, CXX, CFLAGS, etc. pointing to the cross-compiler toolchain. After quite a bit of reading it became obvious that neither CC or LD environment variables wheren't being used to define the linker for the rustc compiler.

As such I've tried passing it through the environment such as CARGO_TARGET_ARMV7_UNKNOWN_LINUX_GNUEABIHF_LINKER="/home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/arm-unknown-linux-gnueabi-gcc" but it ends-up failing at linking time where the host default linker is being used:

      error: could not compile `libc` due to previous error
      
      Caused by:
        process didn't exit successfully: `rustc --crate-name build_script_build /home/spksrc/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.132/build.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debug-assertions=off -C overflow-checks=on --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=c6828ee39fa0c47f -C extra-filename=-c6828ee39fa0c47f --out-dir /tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/libc-c6828ee39fa0c47f -L dependency=/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/deps --cap-lints allow` (exit status: 1)
      error: linking with `cc` failed: exit status: 1
        |
        = note: "cc" "-m64" "/tmp/rustcdK3NnT/symbols.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.0.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.1.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.10.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.11.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.12.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.13.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.14.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.15.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.2.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.3.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.4.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.5.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.6.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.7.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.8.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.build_script_build.e7ddb37a-cgu.9.rcgu.o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2.3i9cjhtqe7j8wwb5.rcgu.o" "-Wl,--as-needed" "-L" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/deps" "-L" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-8f1929c73c3f8167.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-e359d865975ccf21.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-b886fd10c5a7c7c0.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-9d7c322d48daa475.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-08ae1606a951cabe.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-682a81c4b2133b72.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-a73b3512c88de071.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-7b5ec4c918d9f957.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-65c63cf3af0af657.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-4a53f0a2785abc6a.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-868e2d515c28d027.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-394ad2d73aede76a.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-19c77e4dc3dcb87e.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-fb44a42088c9369a.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-61a7402e61a5b0e0.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-63f8356c87a0d0e8.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-a506e577d917828c.rlib" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-fc1fb63210fdafad.rlib" "-Wl,--end-group" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-c21be34a5cae8449.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/spksrc/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/tmp/pip-wheel-cri4pr2f/cryptography_a11930f6101c49faa156270f4708669e/src/rust/target/release/build/pyo3-build-config-0257034bb1a5cea2/build_script_build-0257034bb1a5cea2" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs"
        = note: /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: relocations in generic ELF (EM: 40)
                /usr/bin/ld: /home/spksrc/rustc/spksrc/toolchain/syno-armv7-7.0/work/arm-unknown-linux-gnueabi/bin/../arm-unknown-linux-gnueabi/sysroot/usr/lib/../lib/Scrt1.o: error adding symbols: file in wrong format
                collect2: error: ld returned 1 exit status

My thought is, can/shall I use options such as --global-option=--target="armv7-unknown-linux-gnueabihf" or similar where I can enforce the default target and, most importantly, enforce the default linker?

My current work at integrating rust python wheel cross-compiling is in PR SynoCommunity/spksrc#5435. Although note that github action logs won't provide much as it currently doesn't update the build Docker image until pushed to master.

As mentionned above, guidance would be much appreciated. Thnx in advance.

@messense
Copy link
Member

It's hard to investigate without a working environment, do you have a list of steps to reproduce your build issue locally using docker?

@th0ma7
Copy link
Author

th0ma7 commented Sep 28, 2022

@messense Yes indeed I do, and let me know if you need anything else:

Clone my copy of our spksrc framework repository:

$ git clone https://github.com/th0ma7/spksrc.git
$ cd spksrc
$ git checkout rustc-wheels

Create the updated docker image:

$ docker build - < Dockerfile
...
Successfully built 882a0451f1bb

Access the docker image:

$ docker run -it -v $(pwd):/spksrc 882a0451f1bb /bin/bash

Build a python310 package that will invoke building cryptography==38.0.1

root@2ff602f23b71:/spksrc# cd spk/python310
root@2ff602f23b71:/spksrc/spk/python310/# make arch-armv7-7.0

Options of interest for building are:

  • make arch-aarch64-7.0
  • make arch-x64-7.0
  • make arch-evansport-7.0

To build for every archs and every support Synology DSM Linux version (e.g. 6.1 & 7.0): make all-supported

Note that build logs are being output on the console but are also available in build-<arch>-<DSM-version>.log

@th0ma7
Copy link
Author

th0ma7 commented Oct 1, 2022

Follow to this, my issue seems to be somewhat due to setuptools-rust not behaving properly using a crossenv.

Here's my finding using docker image above (always using a x86_64 host) and also reproduced on a Debian 11 VM. My first thing to try is, is it even cross-compiling using rust? Lets take ripgrep as an easy app to build:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --target=aarch64-unknown-linux-gnu --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

It works! Now let's try that using CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu instead of enforcing --target:

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Still working! So the theory is that I have a working rustc along with a proper recognition of my target gcc as default linker and CARGO* and RUSTUP* environment variables are also recognized and functional. That's a good start :)

More on this first testing can be found SynoCommunity/spksrc#5435 (comment)

Then I've resumed at trying to build cryptography using our spksrc framework. It's rather easy to trigger with PR above, once the build is complete (e.g. make arch-armv7-7.0) you simply have to first setup your PATH to point towards the crossenv first:

$ export PATH=/spksrc/spk/python310/work-aarch64-7.0/crossenv/cross/bin:/spksrc/spk/python310/work-aarch64-7.0/crossenv/cross/bin:$PATH

Then build cryptography from the extracted source:

$ CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
python3 setup.py bdist_wheel --py-limited-api=cp36

Interestingly enough it gets a little farther using RUSTFLAGS such as:

RUSTFLAGS="--target=aarch64-unknown-linux-gnu -Clinker=/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
python3 setup.py bdist_wheel --py-limited-api=cp36

A call using crossenv pip will also result with the same outcome:

CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" \
CARGO_BUILD_TARGET=aarch64-unknown-linux-gnu \
CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" \
PYO3_CROSS_LIB_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/lib/ \
PYO3_CROSS_INCLUDE_DIR=/spksrc/spk/python310/work-aarch64-7.0/install/var/packages/python310/target/include/ \
pip install cryptography==38.0.1

My conclusion is that setuptools-rust is somehow mixing-up things between host & target and not always using the proper linker, often returning to /usr/bin/ld instead of the provided one.

@th0ma7 th0ma7 changed the title Advices/help needed to cross-compile rust python wheels setuptools-rust fails to cross-compile cryptography using crossenv Oct 1, 2022
@messense
Copy link
Member

messense commented Oct 1, 2022

# CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="/spksrc/toolchain/syno-aarch64-7.0/work/aarch64-unknown-linux-gnu/bin/aarch64-unknown-linux-gnu-gcc" CARGO_HOME="/opt/cargo" RUSTUP_HOME="/opt/rustup" cargo install ripgrep --target=aarch64-unknown-linux-gnu --force
# file rg
rg: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, with debug_info, not stripped

Does it still work if you also add all the CFLAGS/CXXFLAGS/LDFLAGS env vars?

@messense
Copy link
Member

messense commented Oct 1, 2022

Looks like this circles back to an issue I posted before? benfogle/crossenv#65

@messense
Copy link
Member

messense commented Oct 1, 2022

In our CI we cleared the LIBRARY_PATH crossenv set via --env LIBRARY_PATH=

python3.9 -m crossenv "/opt/python/cp39-cp39/bin/python3" --cc $TARGET_CC --cxx $TARGET_CXX --sysroot $TARGET_SYSROOT --env LIBRARY_PATH= --manylinux manylinux1 venv

@th0ma7
Copy link
Author

th0ma7 commented Oct 1, 2022

I confirm, editing the crossenv/cross/bin/python3 and removing the LIBRARY_PATH from the extra_envs variable works!!!!

I got a few SSL deprecation warnings, mays have to find a way to pass -Wno-unused-function ... But it works:

# ll dist/cryptography-38.0.1-cp36-abi3-linux_aarch64.whl 
-rw-r--r-- 1 root root 1810579 Oct  1 08:41 dist/cryptography-38.0.1-cp36-abi3-linux_aarch64.whl

@th0ma7
Copy link
Author

th0ma7 commented Oct 1, 2022

@messense is there an intent to include the --env LIBRARY_PATH= as built-in to setuptools-rust or should I apply a workaround on my end?

@messense
Copy link
Member

messense commented Oct 2, 2022

is there an intent to include the --env LIBRARY_PATH= as built-in to setuptools-rust

No, we don't want to add too much assumption in setuptools-rust, bad assumption caused a trouble in the past. It's better to be dealt by user.

Closing as resolved.

@messense messense closed this as completed Oct 2, 2022
@th0ma7
Copy link
Author

th0ma7 commented Oct 2, 2022

@messense having a global option similar to --library-path= or --disable-library-path would be useful and allow not breaking the code while adding extra functionality

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants