CIBW_ENVIRONMENT not evaluated on Linux

[JesseFarebro]

It seems that the environment variables passed through CIBW_ENVIRONMENT aren't evaluated on Linux. From a first look, it seems that as_shell_commands() doesn't use the evaluated environment variables.

https://github.com/joerick/cibuildwheel/blob/7709989603e88d28419bb936c46d1030a0aefaf3/cibuildwheel/linux.py#L145

https://github.com/joerick/cibuildwheel/blob/7709989603e88d28419bb936c46d1030a0aefaf3/cibuildwheel/environment.py#L76-L77

https://github.com/joerick/cibuildwheel/blob/7709989603e88d28419bb936c46d1030a0aefaf3/cibuildwheel/environment.py#L56-L57

This wouldn't be an issue on Windows or macOS as they seem to use as_dictionary which uses the evaluated values.

[joerick]

Hi @JesseFarebro , CIBW_ENVIRONMENT items are evaluated by bash on Linux, not by cibuildwheel directly. Maybe you could share an example of it not working as expected?

[JesseFarebro]

Hi @joerick,

Perhaps this is an issue with Github Actions. I have a minimal reproduction with @mayeut's branch for Github Actions. Here is the repo: https://github.com/JesseFarebro/cibuildwheel-env-repo. Just check the actions tab and you can look at the logs.

I try to forward GITHUB_REF so it's available in the Docker container but if you just search through the logs for GITHUB_REF you can see it passes a blank value to the Docker instance. You can also verify this isn't an issue with the other two platforms.

[JesseFarebro]

On further searching, this does seem to be what's going on. Github Actions doesn't evaluate the YML inside of a shell context (see https://github.community/t5/GitHub-Actions/Set-env-to-command-output/td-p/30322).

I guess I could just define it in the script or perhaps for Github Actions cibuildwheel could evaluate CIBW_ENVIRONMENT?

[joerick]

Those threads look like they might contain solutions, but we've also discussed the possibility of a CIBW_PASSENV option that would let you specify environment variables to passthrough to the docker container. #117 I'd interpret this as a +1 for that feature request!

otherwise, some people have worked around this by using a shell command before invoking cibw, like:

      run: |
        export CIBW_ENVIRONMENT="GITHUB_REF=$GITHUB_REF"
        cibuildwheel --output-dir dist

[YannickJadoul]

Evaluating CIBW_ENVIRONMENT in cibuildwheel (and thus in the outer environment instead of the manylinux docker image's environment) would be problematic in its own way, since it wouldn't allow access to the environment of the docker image (e.g., I use something like CIBW_ENVIRONMENT='ARCH=`uname -i`' to get the architecture of the docker image).

It's a very fine line between having useful abstraction and having a hindersome one :-/