Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup bootstrap dependencies before handoff. #542

Merged
merged 4 commits into from
Aug 29, 2018

Conversation

jsirois
Copy link
Member

@jsirois jsirois commented Aug 20, 2018

Previously we polluted the third party import space with any third
party dependencies of the pex bootstrap code (currently a subset of
setuptools). We now unimport all third_party code in bootstrap and
place bootstrap at the end of sys.path before handing off to user
code to allow their custom versions of overlapping third party code
to be imported.

Added a failing test that the demoted_bootstrap code now fixes.

Fixes #437.

@jsirois
Copy link
Member Author

jsirois commented Aug 20, 2018

Now with full test coverage for all for terminal entry points to executing user code.

It's probably worth noting the new PEX_VERBOSE=1 output:

pex: Please build pex with the subprocess32 module for more reliable requirement installation and interpreter execution.
pex: Found site-library: /usr/lib/site-python
pex: Found site-library: /usr/lib/python2.7/site-packages
pex: Found site extra: /usr/lib/python2.7/site-packages/gtk-2.0
pex: Tainted path element: /usr/lib/python2.7/site-packages
pex: Tainted path element: /usr/lib/python2.7/site-packages/gtk-2.0
pex: Scrubbing from user site: /home/jsirois/.local/lib/python2.7/site-packages
pex: Scrubbing from site-packages: /usr/lib/python2.7/site-packages
pex: Scrubbing from site-packages: /usr/lib/python2.7/site-packages/gtk-2.0
pex: Activating PEX virtual environment from ./dist/pex27: 56.5ms             
pex:     Caching pex 1.4.5: 6.1ms
pex: Bootstrap complete, performing final sys.path modifications...
pex: Un-importing third party bootstrap dependency pkg_resources from /home/jsirois/dev/pantsbuild/jsirois-pex/dist/pex27/.bootstrap
pex: PYTHONPATH contains:
pex:     /home/jsirois/dev/pantsbuild/jsirois-pex/dist/pex27
pex:   * /usr/lib/python27.zip
pex:     /usr/lib/python2.7
pex:     /usr/lib/python2.7/plat-linux2
pex:     /usr/lib/python2.7/lib-tk
pex:   * /usr/lib/python2.7/lib-old
pex:     /usr/lib/python2.7/lib-dynload
pex:     /home/jsirois/.pex/install/certifi-2018.8.13-py2.py3-none-any.whl.f01268216dd6f320b6f45c5903b12f607d03c2c2/certifi-2018.8.13-py2.py3-none-any.whl
pex:     /home/jsirois/.pex/install/pex-1.4.5-py2.py3-none-any.whl.18afaa3965efaec86f4c7dced15bcd2676de2faa/pex-1.4.5-py2.py3-none-any.whl
pex:     /home/jsirois/.pex/install/wheel-0.31.1-py2.py3-none-any.whl.9162405b4c6d47ff554018389d29281b6e4854a6/wheel-0.31.1-py2.py3-none-any.whl
pex:     /home/jsirois/.pex/install/requests-2.19.1-py2.py3-none-any.whl.9c19d1f49965483c8e82326b964ec1050dff6851/requests-2.19.1-py2.py3-none-any.whl
pex:     /home/jsirois/.pex/install/urllib3-1.23-py2.py3-none-any.whl.3630f56fd6bcc8cfd3c568a65834f5b56ebc405e/urllib3-1.23-py2.py3-none-any.whl
pex:     /home/jsirois/.pex/install/idna-2.7-py2.py3-none-any.whl.96a4a6d75966bb0aae4ce9bdd177fc4886b37a07/idna-2.7-py2.py3-none-any.whl
pex:     /home/jsirois/.pex/install/chardet-3.0.4-py2.py3-none-any.whl.c4332e34e38b781695ce775973fe40663558a897/chardet-3.0.4-py2.py3-none-any.whl
pex:     /home/jsirois/.pex/install/setuptools-33.1.1-py2.py3-none-any.whl.d5c7021b0a2ca18f60b7dd7a5b9ffebcb789d43b/setuptools-33.1.1-py2.py3-none-any.whl
pex:   * /home/jsirois/dev/pantsbuild/jsirois-pex/dist/pex27/.bootstrap
pex:   * - paths that do not exist or will be imported via zipimport

pex/pex.py Outdated
@@ -415,6 +410,40 @@ def _execute(self):
TRACER.log('No entry point specified, dropping into interpreter')
return self.execute_interpreter()

@classmethod
@contextmanager
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any point to keeping this as a contextmanager if it doesn't snapshot and restore prior sys.path state? as a caller, I'd expect the contextmgr to undo any mutations vs purely apply them and do nothing upon context block exit.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope. I did start off re-jiggering in finally until I realized it was not needed. I'll fix. The patch_sys has this problem too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth another look - I stripped the contextmanagers from this class. Only one tried to restore anything (patch_pkg_resources) and the restoration was not needed for the pex handling of teardown.

@jsirois
Copy link
Member Author

jsirois commented Aug 24, 2018

OK - rebased past conflicts in master. PTAL at 457bc90.

Previously we polluted the third party import space with any third
party dependencies of the pex bootstrap code (currently a subset of
setuptools). We now unimport all third_party code in bootstrap and
place bootstrap at the end of `sys.path` before handing off to user
code to allow their custom versions of overlapping third party code
to be imported.

Added a failing test that the demoted_bootstrap code now fixes.

Fixes pex-tool#437.
We had a few of this operating in the leadup to hand-off to user
code.
@jsirois
Copy link
Member Author

jsirois commented Aug 28, 2018

Another rebase past conflicts in master @ a972cb2. I'll submit by end of day unless there are objections.

@jsirois jsirois merged commit 8f9c8dd into pex-tool:master Aug 29, 2018
@jsirois jsirois deleted the issues/437 branch August 29, 2018 03:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

pex shadows pkg_resources, which can break newer versions of setuptools
2 participants