Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for selecting via Python build CPU architecture #1149

Closed
tr00st opened this issue Jan 25, 2019 · 9 comments
Closed

Support for selecting via Python build CPU architecture #1149

tr00st opened this issue Jan 25, 2019 · 9 comments
Labels
area:configuration feature:new something does not exist yet, but should help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted. needs:discussion It's not quite clear if and how this should be done

Comments

@tr00st
Copy link

tr00st commented Jan 25, 2019

Would it be feasible to allow switching of Python environments based on CPU architecture?

I've been recently attempting to arrange builds of binary wheel files using Tox to automate the environment switching. This is particularly frustrating on Windows, as getting a working VC++ toolchain is a non-trivial exercise. To deal with this, I'm deploying scripts to our CI server that builds wheels of any binary packages we use, and uploads them to an isolated pypiserver instance. Tox currently works well for iterating over the versions available, but I've been unable to find a way to

Given the "py" launcher tracks CPU architecture, I've been able to come up with a way of patching this support in for Windows (see tr00st@b7dc030) - the patch I've made means a user could set up a set of environments, eg at worst:

[testenv:py35-x86]
basepython = python3.5-32
[testenv:py36-x86]
basepython = python3.6-32
[testenv:py37-x86]
basepython = python3.7-32
[testenv:py35-x64]
basepython = python3.5-64
[testenv:py36-x64]
basepython = python3.6-64
[testenv:py37-x64]
basepython = python3.7-64

This is more than enough for my scenario, but I'm aware this wouldn't solve the issue for Linux users - hence I'm holding off a PR for now. Any idea if this would be desirable, and if so how one might go about this on linux? (things that come to mind are running the python interpreter with a quick one-liner or using "file" package to inspect any potential python executable).

Happy to put a PR through if this is desirable as-is, but thought it would be worth discussing any other possible solutions.

@obestwalter obestwalter added feature:new something does not exist yet, but should area:configuration needs:discussion It's not quite clear if and how this should be done labels Jan 25, 2019
@gaborbernat
Copy link
Member

gaborbernat commented May 2, 2019

I think the main problem is that there's no standard now for how to discover these additional targets. Any suggestions? It's fine if it's a Windows-only feature only though in which case yeah let's do this. I would have a basepython_architecture in this case though. On Linux/Mac x64 should be skipped.

@gaborbernat gaborbernat added the help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted. label May 2, 2019
@mrh1997
Copy link

mrh1997 commented Feb 14, 2020

Proposal:

  • mac: all pythons are 64bit as far as I now => ignore all environments with basepython_architecture = 32
  • linux: all pythons are of the same architecture as the distributions architecture => ignore all environments which explicitly enforce the other architecture.

Admittedly this is not a 100% solution, as it is possible to manually create 32bit binaries on 64bit linuxes (as on windows), but reality is different. On windows both architectures are explicitly distributed for 64bit windows systems, while on linux you can only install binaries for the same architecture as your system uses.

@gaborbernat
Copy link
Member

This is now supported by virtualenv; we'll probably switch to that part of tox 4 rewrite.

@gaborbernat
Copy link
Member

My available efforts at the moment are aimed at fixing this as part of #1394, but that probably will take a while (ETA September).

@jayvdb
Copy link

jayvdb commented Oct 20, 2020

While waiting for tox 4, this can be done by using https://github.com/jayvdb/tox-add-factor

@schlamar
Copy link
Contributor

@jayvdb I don't think your proposal is correct. Please note that this feature request wants to allow different Python architectures on one host (for example the environments Python 3.7 32bit and Python 3.7 64bit on the same Windows x64 host). Your proposed plugin handles different architectures on different hosts AFAIS.

@schlamar
Copy link
Contributor

schlamar commented Oct 21, 2020

For now, I handle this with a small plugin (so I don't have to patch tox).

import subprocess

import pluggy

hookimpl = pluggy.HookimplMarker('tox')

    
@hookimpl
def tox_get_python_executable(envconfig):
    basepython = envconfig.basepython
    if not basepython.startswith('py '):
        return None

    cmd = basepython.split() + ['-c', 'import sys; print(sys.executable)']
    try:
        path = subprocess.check_output(cmd)
    except (OSError, subprocess.CalledProcessError):
        return None
    return path.decode('ascii').strip()

And configure my environments:

[testenv:py37]
basepython = py -3.7-32

[testenv:py37-64]
basepython = py -3.7-64

Feel free to upload this on PyPI if someone is interested.

@ionelmc
Copy link

ionelmc commented Oct 22, 2020

Note that you can use this trick:

[testenv]
basepython =
    py37: {env:TOXPYTHON:python3.7}
    py38: {env:TOXPYTHON:python3.8}
    ...

And then set TOXPYTHON in the environment to the preferred arch python.exe.

@gaborbernat
Copy link
Member

This is now possible by virtualenv understanding these, and users can set basepython accordingly (which is passed through virtualenv) https://virtualenv.pypa.io/en/latest/user_guide.html#python-discovery

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:configuration feature:new something does not exist yet, but should help:wanted Issues that have been acknowledged, a solution determined and a PR might likely be accepted. needs:discussion It's not quite clear if and how this should be done
Projects
None yet
Development

No branches or pull requests

7 participants