Bug 1732946: Vendor pip/setuptools/wheel instead of installing into v…

…env r=ahal

Now that are prioritizing system over virtualenv site-packages, the
system `pip` is sometimes being used instead.
This is causing issues when the system pip is set up in a
distro-specific way, such as when "debundled":
https://github.com/pypa/pip/blob/9.0.1/pip/_vendor/__init__.py#L53-L61

However, if we vendor `pip`, `setuptools` and `wheel`, and ensure that
they're prioritized in the `sys.path` before anything is imported from
the system, then we can ensure that we're using a modern `pip` _and_
sidestep system-specific pip weirdness.

Note that `pip-compile`'s `--allow-unsafe` flag is not as dangerous as
it sounds.
There's confusion among maintainers about its origin:
jazzband/pip-tools#522
Additionally, it's going to be enabled by default in a future
`pip-tools` release. So, it's not scary for us to embrace here.

Also, heads up that the "pip outdated warning" no longer needs
to be manually silenced, since pip avoids that code path when
not running from an "installed" context.

Differential Revision: https://phabricator.services.mozilla.com/D127182

build/common_virtualenv_packages.txt

@@ -88,6 +88,7 @@ vendored:third_party/python/mozilla_version
 vendored:third_party/python/multidict
 vendored:third_party/python/packaging
 vendored:third_party/python/pathspec
+vendored:third_party/python/pip
 vendored:third_party/python/pip_tools
 vendored:third_party/python/ply
 vendored:third_party/python/pyasn1
@@ -103,6 +104,7 @@ vendored:third_party/python/requests
 vendored:third_party/python/requests_unixsocket
 vendored:third_party/python/responses
 vendored:third_party/python/rsa
+vendored:third_party/python/setuptools
 vendored:third_party/python/sentry_sdk
 vendored:third_party/python/six
 vendored:third_party/python/slugid
@@ -112,6 +114,7 @@ vendored:third_party/python/taskcluster_urls
 vendored:third_party/python/typing_extensions
 vendored:third_party/python/urllib3
 vendored:third_party/python/voluptuous
+vendored:third_party/python/wheel
 vendored:third_party/python/yamllint
 vendored:third_party/python/yarl
 vendored:third_party/python/zipp

python/mach/mach/virtualenv.py

@@ -248,8 +248,8 @@ def up_to_date(self, skip_pip_package_check=False):
                 return False
 
         if not skip_pip_package_check:
-            pip = os.path.join(self.bin_path, "pip")
-            package_result = env_requirements.validate_environment_packages([pip])
+            pip = [self.python_path, "-m", "pip"]
+            package_result = env_requirements.validate_environment_packages(pip)
             if not package_result.has_all_packages:
                 return False
 
@@ -302,7 +302,7 @@ def create(self):
             # world and search for or download a newer version of pip,
             # setuptools, or wheel. This is bad for security, reproducibility,
             # and speed.
-            "--no-download",
+            "--no-seed",
             self.virtualenv_root,
         ]
 
@@ -314,7 +314,6 @@ def create(self):
                 % (self.virtualenv_root, result)
             )
 
-        self._disable_pip_outdated_warning()
         return self.virtualenv_root
 
     @functools.lru_cache(maxsize=None)
@@ -448,40 +447,6 @@ def install_pip_requirements(self, path, require_hashes=True, quiet=False):
 
         return self._run_pip(args, stderr=subprocess.STDOUT)
 
-    def _disable_pip_outdated_warning(self):
-        """Disables the pip outdated warning by changing pip's 'installer'
-
-        "pip" has behaviour to ensure that it doesn't print it's "outdated"
-        warning if it's part of a Linux distro package. This is because
-        Linux distros generally have a slightly out-of-date pip package
-        that they know to be stable, and users aren't always able to
-        (or want to) update it.
-
-        This behaviour works by checking if the "pip" installer
-        (encoded in the dist-info/INSTALLER file) is "pip" itself,
-        or a different value (e.g.: a distro).
-
-        We can take advantage of this behaviour by telling pip
-        that it was installed by "mach", so it won't print the
-        warning.
-
-        https://github.com/pypa/pip/blob/5ee933aab81273da3691c97f2a6e7016ecbe0ef9/src/pip/_internal/self_outdated_check.py#L100-L101 # noqa F401
-        """
-        site_packages = self._site_packages_dir()
-        pip_dist_info = next(
-            (
-                file
-                for file in os.listdir(site_packages)
-                if file.startswith("pip-") and file.endswith(".dist-info")
-            ),
-            None,
-        )
-        if not pip_dist_info:
-            raise Exception("Failed to find pip dist-info in new virtualenv")
-
-        with open(os.path.join(site_packages, pip_dist_info, "INSTALLER"), "w") as file:
-            file.write("mach")
-
     def _run_pip(self, args, **kwargs):
         kwargs.setdefault("check", True)
 
@@ -495,9 +460,9 @@ def _run_pip(self, args, **kwargs):
         # force the virtualenv's interpreter to be used and all is well.
         # It /might/ be possible to cheat and set sys.executable to
         # self.python_path. However, this seems more risk than it's worth.
-        pip = os.path.join(self.bin_path, "pip")
+        pip = [self.python_path, "-m", "pip"]
         return subprocess.run(
-            [pip] + args, cwd=self.topsrcdir, env=env, universal_newlines=True, **kwargs
+            pip + args, cwd=self.topsrcdir, env=env, universal_newlines=True, **kwargs
         )
 
     def _site_packages_dir(self):

python/mozbuild/mozbuild/test/test_vendor.py

@@ -27,10 +27,12 @@ def test_up_to_date_vendor():
         ) as file:
             # Since VendorPython thinks "work_dir" is the topsrcdir,
             # it will use its associated virtualenv and package configuration.
-            # Since it uses "pip-tools" within, and "pip-tools" needs
-            # the "Click" library, we need to make them available.
+            # Add `pip-tools` and its dependencies.
             file.write("vendored:third_party/python/Click\n")
+            file.write("vendored:third_party/python/pip\n")
             file.write("vendored:third_party/python/pip_tools\n")
+            file.write("vendored:third_party/python/setuptools\n")
+            file.write("vendored:third_party/python/wheel\n")
 
         # Copy existing "third_party/python/" vendored files
         existing_vendored = os.path.join(topsrcdir, "third_party", "python")

python/mozbuild/mozbuild/vendor/vendor_python.py

@@ -37,7 +37,8 @@ def vendor(self, keep_extra_files=False):
             # of transitive dependencies aren't implicitly changed.
             shutil.copy(requirements, tmp_requirements_absolute)
 
-            # resolve the dependencies and update requirements.txt
+            # resolve the dependencies and update requirements.txt.
+            # "--allow-unsafe" is required to vendor pip and setuptools.
             subprocess.check_output(
                 [
                     self.virtualenv_manager.python_path,
@@ -50,6 +51,7 @@ def vendor(self, keep_extra_files=False):
                     "--output-file",
                     tmp_requirements_absolute,
                     "--generate-hashes",
+                    "--allow-unsafe",
                 ],
                 # Run pip-compile from within the temporary directory so that the "via"
                 # annotations don't have the non-deterministic temporary path in them.

python/mozperftest/mozperftest/tests/support.py

@@ -49,9 +49,8 @@ def get_binary_path(*args):
         return ""
 
     def run_pip(args):
-        pip = Path(sys.executable).parent / "pip"
         subprocess.check_call(
-            [str(pip)] + args,
+            [sys.executable, "-m", "pip"] + args,
             stderr=subprocess.STDOUT,
             cwd=config.topsrcdir,
             universal_newlines=True,

third_party/python/pip/pip-21.2.4.dist-info/LICENSE.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2008-2021 The pip developers (see AUTHORS.txt file)
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

third_party/python/pip/pip-21.2.4.dist-info/METADATA

@@ -0,0 +1,92 @@
+Metadata-Version: 2.1
+Name: pip
+Version: 21.2.4
+Summary: The PyPA recommended tool for installing Python packages.
+Home-page: https://pip.pypa.io/
+Author: The pip developers
+Author-email: distutils-sig@python.org
+License: MIT
+Project-URL: Documentation, https://pip.pypa.io
+Project-URL: Source, https://github.com/pypa/pip
+Project-URL: Changelog, https://pip.pypa.io/en/stable/news/
+Platform: UNKNOWN
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Software Development :: Build Tools
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: Implementation :: CPython
+Classifier: Programming Language :: Python :: Implementation :: PyPy
+Requires-Python: >=3.6
+License-File: LICENSE.txt
+
+pip - The Python Package Installer
+==================================
+
+.. image:: https://img.shields.io/pypi/v/pip.svg
+   :target: https://pypi.org/project/pip/
+
+.. image:: https://readthedocs.org/projects/pip/badge/?version=latest
+   :target: https://pip.pypa.io/en/latest
+
+pip is the `package installer`_ for Python. You can use pip to install packages from the `Python Package Index`_ and other indexes.
+
+Please take a look at our documentation for how to install and use pip:
+
+* `Installation`_
+* `Usage`_
+
+We release updates regularly, with a new version every 3 months. Find more details in our documentation:
+
+* `Release notes`_
+* `Release process`_
+
+In pip 20.3, we've `made a big improvement to the heart of pip`_; `learn more`_. We want your input, so `sign up for our user experience research studies`_ to help us do it right.
+
+**Note**: pip 21.0, in January 2021, removed Python 2 support, per pip's `Python 2 support policy`_. Please migrate to Python 3.
+
+If you find bugs, need help, or want to talk to the developers, please use our mailing lists or chat rooms:
+
+* `Issue tracking`_
+* `Discourse channel`_
+* `User IRC`_
+
+If you want to get involved head over to GitHub to get the source code, look at our development documentation and feel free to jump on the developer mailing lists and chat rooms:
+
+* `GitHub page`_
+* `Development documentation`_
+* `Development mailing list`_
+* `Development IRC`_
+
+Code of Conduct
+---------------
+
+Everyone interacting in the pip project's codebases, issue trackers, chat
+rooms, and mailing lists is expected to follow the `PSF Code of Conduct`_.
+
+.. _package installer: https://packaging.python.org/guides/tool-recommendations/
+.. _Python Package Index: https://pypi.org
+.. _Installation: https://pip.pypa.io/en/stable/installation/
+.. _Usage: https://pip.pypa.io/en/stable/
+.. _Release notes: https://pip.pypa.io/en/stable/news.html
+.. _Release process: https://pip.pypa.io/en/latest/development/release-process/
+.. _GitHub page: https://github.com/pypa/pip
+.. _Development documentation: https://pip.pypa.io/en/latest/development
+.. _made a big improvement to the heart of pip: https://pyfound.blogspot.com/2020/11/pip-20-3-new-resolver.html
+.. _learn more: https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-3-2020
+.. _sign up for our user experience research studies: https://pyfound.blogspot.com/2020/03/new-pip-resolver-to-roll-out-this-year.html
+.. _Python 2 support policy: https://pip.pypa.io/en/latest/development/release-process/#python-2-support
+.. _Issue tracking: https://github.com/pypa/pip/issues
+.. _Discourse channel: https://discuss.python.org/c/packaging
+.. _Development mailing list: https://mail.python.org/mailman3/lists/distutils-sig.python.org/
+.. _User IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa
+.. _Development IRC: https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:+6697/pypa-dev
+.. _PSF Code of Conduct: https://github.com/pypa/.github/blob/main/CODE_OF_CONDUCT.md
+
+