Linux build can't access host environment variables

[jonathanunderwood]

The environment variables for commands run on linux and macos are inconsistent. On macos, the host environment variables are available to CIBW_TEST_COMMAND, whereas on linux they're not.

For an example, see: https://travis-ci.org/python-lz4/python-lz4/builds/471284698 (compare py27 jobs).

On macos:

CIBW_BEFORE_BUILD=pip install -U pip
TRAVIS_ARCH=amd64
rvm_bin_path=/Users/travis/.rvm/bin
TRAVIS_FILTERED=redirect_io
NVM_CD_FLAGS=
HAS_JOSH_K_SEAL_OF_APPROVAL=true
GEM_HOME=/Users/travis/.rvm/gems/ruby-2.4.3
TERM=xterm
TRAVIS_TEST_RESULT=
SHELL=/bin/bash
IRBRC=/Users/travis/.rvm/rubies/ruby-2.4.3/.irbrc
TMPDIR=/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/
TRAVIS_COMMIT=7a444a879a2bde6eedcbe78f4b3d7974f0c39da2
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.SdabHcQ8xD/Render
TRAVIS_OS_NAME=osx
TRAVIS_APT_PROXY=http://build-cache.travisci.net
TRAVIS_JOB_NAME=osx-py27
TRAVIS_INTERNAL_RUBY_REGEX=^ruby-(2\.[0-4]\.[0-9]|1\.9\.3)
MY_RUBY_HOME=/Users/travis/.rvm/rubies/ruby-2.4.3
TRAVIS_ROOT=/
CIBW_TEST_COMMAND=env && pytest --cov=lz4 {project}/tests && cp .coverage {project} && pushd {project} && codecov --required; coveralls
TRAVIS_TIMER_ID=0ac79408
LC_ALL=en_US.UTF-8
ANSI_GREEN=\033[32;1m
USER=travis
TWINE_PASSWORD=[secure]
NVM_DIR=/Users/travis/.nvm
_system_type=Darwin
TRAVIS_LANGUAGE=generic
TRAVIS_INFRA=macstadium
ANSI_RESET=\033[0m
rvm_path=/Users/travis/.rvm
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.e40qkbfHw7/Listeners
TRAVIS_DIST=notset
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
TRAVIS=true
ANSI_YELLOW=\033[33;1m
TRAVIS_REPO_SLUG=python-lz4/python-lz4
PAGER=cat
TRAVIS_BUILD_STAGE_NAME=
TRAVIS_COMMIT_MESSAGE=Print out env ahead of testing
TRAVIS_PULL_REQUEST=false
TRAVIS_CMD=cibuildwheel --output-dir dist
rvm_prefix=/Users/travis
PATH=/tmp/cibw_bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/Users/travis/.rvm/gems/ruby-2.4.3/bin:/Users/travis/.rvm/gems/ruby-2.4.3@global/bin:/Users/travis/.rvm/rubies/ruby-2.4.3/bin:/Users/travis/.rvm/bin:/Users/travis/bin:/Users/travis/.local/bin:/Users/travis/.nvm/versions/node/v6.14.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
TRAVIS_OSX_IMAGE=
TRAVIS_PULL_REQUEST_SHA=
TRAVIS_JOB_WEB_URL=https://travis-ci.org/python-lz4/python-lz4/jobs/471284704
TRAVIS_TMPDIR=/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/tmp.cACKT12w
_=/usr/bin/env
TRAVIS_BUILD_WEB_URL=https://travis-ci.org/python-lz4/python-lz4/builds/471284698
PWD=/Users/travis
CONTINUOUS_INTEGRATION=true
LANG=en_US.UTF-8
CIBUILDWHEEL=1
_system_arch=x86_64
XPC_FLAGS=0x0
TRAVIS_ENABLE_INFRA_DETECTION=true
_system_version=10.13
TRAVIS_SUDO=true
TRAVIS_TAG=
TRAVIS_ALLOW_FAILURE=false
XPC_SERVICE_NAME=0
TRAVIS_HOME=/Users/travis
PIP=pip
TRAVIS_INIT=notset
rvm_version=1.29.3 (latest)
rvm_pretty_print_flag=auto
TRAVIS_EVENT_TYPE=push
TRAVIS_JOB_NUMBER=904.6
SHLVL=2
PS4=+
HOME=/Users/travis
ANSI_CLEAR=\033[0K
TRAVIS_TIMER_START_TIME=1545467497138088000
CI=true
TRAVIS_BUILD_ID=471284698
LOGNAME=travis
GEM_PATH=/Users/travis/.rvm/gems/ruby-2.4.3:/Users/travis/.rvm/gems/ruby-2.4.3@global
TRAVIS_PULL_REQUEST_SLUG=
TRAVIS_SECURE_ENV_VARS=true
DEBIAN_FRONTEND=noninteractive
NVM_BIN=/Users/travis/.nvm/versions/node/v6.14.3/bin
TRAVIS_APP_HOST=build.travis-ci.org
GIT_ASKPASS=echo
TRAVIS_BRANCH=simplify-travis
DISPLAY=/private/tmp/com.apple.launchd.w4V0RVJsYi/org.macosforge.xquartz:0
TRAVIS_COMMIT_RANGE=bc87e556fc52...7a444a879a2b
CIBW_BUILD=cp27-*
TRAVIS_PULL_REQUEST_BRANCH=
TRAVIS_JOB_ID=471284704
ANSI_RED=\033[31;1m
RUBY_VERSION=ruby-2.4.3
_system_name=OSX
TRAVIS_BUILD_DIR=/Users/travis/build/python-lz4/python-lz4
TRAVIS_BUILD_NUMBER=904
CIBW_TEST_REQUIRES=pytest!=3.3.0 pytest-cov codecov psutil coveralls
BASH_FUNC_shell_session_update%%=() {  :
}

On linux:

HOSTNAME=594310d61a80
SSL_CERT_FILE=/opt/_internal/certs.pem
LC_ALL=en_US.UTF-8
LD_LIBRARY_PATH=/opt/rh/devtoolset-2/root/usr/lib64:/opt/rh/devtoolset-2/root/usr/lib:/usr/local/lib64:/usr/local/lib
PATH=/opt/python/cp27-cp27m/bin:/opt/rh/devtoolset-2/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
PWD=/root
LANG=en_US.UTF-8
CIBUILDWHEEL=1
HOME=/root
SHLVL=2
LANGUAGE=en_US.UTF-8
PKG_CONFIG_PATH=/usr/local/lib/pkgconfig

[jonathanunderwood]

Of course, this is because on linux, the test environment is in the docker container. I think the right way to tackle this is during the docker create, where the full environment should be passed in (with multiple -e commands, or with an env file generated from the environment). WDYT?

[YannickJadoul]

I would agree in spirit. I have noticed this asymmetry before, and it seems to go into the purpose of cibuildwheel of trying to hide all of this nasty details (of docker containers, different architectures, different Python versions and installation of them, ...)

However, on a technical/practical level, I don't think it's that easy, actually. Lots of these environment variables wouldn't make sense/would be wrong within the docker, no?
I'm for example thinking of GEM_HOME, SSH_AUTH_SOCK, and even TRAVIS_ARCH (because the we run both 32- and 64-bit docker images on the same Travis architecture).
How would you cope with these?

I believe this is the main reason why the CIBW_ENVIRONMENT option was introduced, to have a cross-platform way of specifying the necessary environment variables. However, I don't think it is possible with these to 'copy' variables for the host? :-/

[jonathanunderwood]

@YannickJadoul I think the most important thing is to be as consistent as possible across platforms. Your concerns are good ones though. I think an approach that could work would be to introduce an environment variable CIBW_PASSENV that would contain a list of vars to pass through to the build context. Similar to how tox does it.

I believe this is the main reason why the CIBW_ENVIRONMENT option was introduced, to have a cross-platform way of specifying the necessary environment variables. However, I don't think it is possible with these to 'copy' variables for the host? :-/

Yes; I tried that: it doesn't work.

[YannickJadoul]

Yes; I tried that: it doesn't work.

OK, then I agree with the principle that we should find a (semi-)transparant way of transferring environment variables.

Maybe we can introduce a way/notation in the CIBW_ENVIRONMENT to access the ones from the host?

Another thing I just thought of, would be something similar to the /host directory, but then for the environment variables. For example, a mapping from GEM_HOME to HOST_GEM_HOME (or CIBW_HOST_GEM_HOME)? This would make it 1) easily accessible by default, and 2) would allow for CIBW_ENVIRONMENT="GEM_HOME=CIBW_HOST_GEM_HOME" to achieve the previous solution?

[jonathanunderwood]

I still think re-using the PASSENV approach is cleaner, and more familiar to users.

[joerick]

I'm not familiar with either approaches, so coming to this cold. Initially I prefer the CIBW_HOST_* approach, since it is quite obvious where an env var comes from. However, I think the big downside is that there would have to be special cases to use these variables inside the container, which seems a bit leaky abstraction. So I'm wondering if we can do a PASSENV approach with a default that covers 90% of use cases, something like CI* TRAVIS_*. Or, can we import all the host variables, except ones that are already defined in the container, like PATH.

[henryiii]

This would be really nice to have to make pyproject.toml easier (see #799 (comment)); how about just going with the explicit and simple option, CIBW_PASSENV?

[tool.cibuildwheel.linux]
passenv = ["MY_VAR", "GITHUB_REF"]

It would be ignored on macOS/Windows, and would pass the listed variables in for Linux.

[joerick]

This was implemented in #914.

[mcleantom]

This would be really nice to have to make pyproject.toml easier (see #799 (comment)); how about just going with the explicit and simple option, CIBW_PASSENV?

[tool.cibuildwheel.linux]
passenv = ["MY_VAR", "GITHUB_REF"]

It would be ignored on macOS/Windows, and would pass the listed variables in for Linux.

Does cibw pass windows environment variables automatically? In the docs I saw CIBW_ENVIRONMENT_PASS_LINUX (https://cibuildwheel.pypa.io/en/stable/options/#environment-pass) but I couldnt see one for windows. Could I do:

[tool.cibuildwheel]
passenv = ["MY_VAR", "GITHUB_REF"]

To pass environment variables for both windows and linux?

[Czaki]

For windows and macOS we build on the same machine as the command is triggered, so all environment variable are automatically passed.

On Linux, we use a docker container, so we need to copy variables into the container. so the settings is relevant only to linux.