Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup isolated environment creation #11257

Merged
merged 1 commit into from
Jul 15, 2022

Conversation

pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented Jul 14, 2022

Based on my tests, on an M1 MacBook Air, this improved pip install ../furo --no-deps from 12-13s to 8-9s. Both things were run interlaced, 4 times, ignoring the first run of each.

I imagine this translates to full test suite runs as well, but the numbers I have are skewed since doing a test run makes my laptop heat up, resulting in signficant-enough heat-related throttling that would skew the results.

@pradyunsg pradyunsg added the type: enhancement Improvements to functionality label Jul 14, 2022
@pradyunsg pradyunsg force-pushed the speedup-environment-creation branch from c0f0739 to ea9cdf8 Compare July 14, 2022 12:50
@pradyunsg pradyunsg force-pushed the speedup-environment-creation branch 3 times, most recently from ab9d076 to 1493b12 Compare July 15, 2022 07:12
@pradyunsg pradyunsg marked this pull request as ready for review July 15, 2022 07:46
@pradyunsg
Copy link
Member Author

pradyunsg commented Jul 15, 2022

@pfmoore, @graingert and @sbidoul I've incorporated your suggestions, added a news file and squashed the commits. Please take a look again! :)

Copy link
Member

@pfmoore pfmoore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One typo and one more substantial point that we might be able to improve. But I'm fine with this as it stands.

src/pip/_internal/build_env.py Outdated Show resolved Hide resolved
# Return the current instance if `source` is not a directory. We can't build
# a zip from this, and it likely means the instance is already standalone.
# Return the current instance if `source` is not a directory. It likely
# means that this copy of pip is already standalone.
Copy link
Member

@pfmoore pfmoore Jul 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the runner, pip_location will point to the current source. So this test won't work as expected, and we'll create a second runner hook. Can we do something by looking at pip.__spec__?

In practice, I don't think having multiple runner hooks is a big issue, so I'm OK if we don't worry too much (although maybe note that we've ignored the issue in a comment, to help people looking at this code in future 😉)

Copy link
Member Author

@pradyunsg pradyunsg Jul 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well... this check is still useful for the zipapp case.

I'd prefer to defer the reuse of the runner script for a follow-up though. We should be able to make the runner script into a module and call that from the sources directly (I consider inlining code in a string of a module to be a code smell) -- avoiding the need to create a runner script on environment creation in the first place. :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've filed #11262, which does this FWIW.

Instead of creating a zip file from the current pip's sources, add the
current copy of pip, to the build environment's interpreter's import
system using `sys.meta_path`. This avoids the overhead of creating the
zipfile, allows us to use the current pip's sources as-is,
meaningfully reduces the size of the build environment and
speeds up the creation of the build environment.
@pradyunsg pradyunsg force-pushed the speedup-environment-creation branch from 1493b12 to d36bd5a Compare July 15, 2022 09:09
@pradyunsg pradyunsg merged commit e975318 into pypa:main Jul 15, 2022
@pradyunsg pradyunsg deleted the speedup-environment-creation branch July 15, 2022 10:09
inmantaci pushed a commit to inmanta/inmanta-core that referenced this pull request Jul 21, 2022
Bumps [pip](https://github.com/pypa/pip) from 22.1.2 to 22.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>22.2 (2022-07-21)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>Remove the <code>html5lib</code> deprecated feature flag. (<code>[#10825](pypa/pip#10825) &lt;https://github.com/pypa/pip/issues/10825&gt;</code>_)</li>
<li>Remove <code>--use-deprecated=backtrack-on-build-failures</code>. (<code>[#11241](pypa/pip#11241) &lt;https://github.com/pypa/pip/issues/11241&gt;</code>_)</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p>Add support to use <code>truststore &lt;https://pypi.org/project/truststore/&gt;</code>_ as an
alternative SSL certificate verification backend. The backend can be enabled on Python
3.10 and later by installing <code>truststore</code> into the environment, and adding the
<code>--use-feature=truststore</code> flag to various pip commands.</p>
<p><code>truststore</code> differs from the current default verification backend (provided by
<code>certifi</code>) in it uses the operating system’s trust store, which can be better
controlled and augmented to better support non-standard certificates. Depending on
feedback, pip may switch to this as the default certificate verification backend in
the future. (<code>[#11082](pypa/pip#11082) &lt;https://github.com/pypa/pip/issues/11082&gt;</code>_)</p>
</li>
<li>
<p>Add <code>--dry-run</code> option to <code>pip install</code>, to let it print what it would install but
not actually change anything in the target environment. (<code>[#11096](pypa/pip#11096) &lt;https://github.com/pypa/pip/issues/11096&gt;</code>_)</p>
</li>
<li>
<p>Record in wheel cache entries the URL of the original artifact that was downloaded
to build the cached wheels. The record is named <code>origin.json</code> and uses the PEP 610
Direct URL format. (<code>[#11137](pypa/pip#11137) &lt;https://github.com/pypa/pip/issues/11137&gt;</code>_)</p>
</li>
<li>
<p>Support <code>PEP 691 &lt;https://peps.python.org/pep-0691/&gt;</code><em>. (<code>[#11158](pypa/pip#11158) &lt;https://github.com/pypa/pip/issues/11158&gt;</code></em>)</p>
</li>
<li>
<p>pip's deprecation warnings now subclass the built-in <code>DeprecationWarning</code>, and
can be suppressed by running the Python interpreter with
<code>-W ignore::DeprecationWarning</code>. (<code>[#11225](pypa/pip#11225) &lt;https://github.com/pypa/pip/issues/11225&gt;</code>_)</p>
</li>
<li>
<p>Add <code>pip inspect</code> command to obtain the list of installed distributions and other
information about the Python environment, in JSON format. (<code>[#11245](pypa/pip#11245) &lt;https://github.com/pypa/pip/issues/11245&gt;</code>_)</p>
</li>
<li>
<p>Significantly speed up isolated environment creation, by using the same
sources for pip instead of creating a standalone installation for each
environment. (<code>[#11257](pypa/pip#11257) &lt;https://github.com/pypa/pip/issues/11257&gt;</code>_)</p>
</li>
<li>
<p>Add an experimental <code>--report</code> option to the install command to generate a JSON report
of what was installed. In combination with <code>--dry-run</code> and <code>--ignore-installed</code> it
can be used to resolve the requirements. (<code>[#53](pypa/pip#53) &lt;https://github.com/pypa/pip/issues/53&gt;</code>_)</p>
</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Fix <code>pip install --pre</code> for packages with pre-release build dependencies defined
both in <code>pyproject.toml</code>'s <code>build-system.requires</code> and <code>setup.py</code>'s
<code>setup_requires</code>. (<code>[#10222](pypa/pip#10222) &lt;https://github.com/pypa/pip/issues/10222&gt;</code>_)</li>
<li>When pip rewrites the shebang line in a script during wheel installation,
update the hash and size in the corresponding <code>RECORD</code> file entry. (<code>[#10744](pypa/pip#10744) &lt;https://github.com/pypa/pip/issues/10744&gt;</code>_)</li>
<li>Do not consider a <code>.dist-info</code> directory found inside a wheel-like zip file
as metadata for an installed distribution. A package in a wheel is (by</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/8e7e76e60f4e115ea1201bee2f176377a718fce1"><code>8e7e76e</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/b6f6a94e36f10a4535ea5bbdc6b351f62003eede"><code>b6f6a94</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/790725aca3f60c745e33827a6079d9600da373d8"><code>790725a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11274">#11274</a> from sbidoul/install-report-note-sbi</li>
<li><a href="https://github.com/pypa/pip/commit/d4b9e187aa7cc5ab14b2339f6171f7f2ea6504e9"><code>d4b9e18</code></a> Add clarifications to the installation report documentation</li>
<li><a href="https://github.com/pypa/pip/commit/b1a01ef762a78af1194958a1c874015eaf81fd04"><code>b1a01ef</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11265">#11265</a> from finnagin/main</li>
<li><a href="https://github.com/pypa/pip/commit/48bcb0a4ccd30a9d00e58fe58827772e307a7e39"><code>48bcb0a</code></a> reformat to pass pre-commit check</li>
<li><a href="https://github.com/pypa/pip/commit/a7c1fe3bff5655393018c53b448b669b3525515b"><code>a7c1fe3</code></a> Remove utc fixture from tests</li>
<li><a href="https://github.com/pypa/pip/commit/0c574f72905185d62bcca741c813df9bae1d9282"><code>0c574f7</code></a> Remove time import</li>
<li><a href="https://github.com/pypa/pip/commit/246fef19149eea893f1cf3efd53f9b17c94c952f"><code>246fef1</code></a> Remove utc fixture</li>
<li><a href="https://github.com/pypa/pip/commit/c9cb7f4629bdd8c61b792feff6dacb1d2e848d57"><code>c9cb7f4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11270">#11270</a> from uranusjr/upgrade-pre-commit-hooks</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.1.2...22.2">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.1.2&new-version=22.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
inmantaci pushed a commit to inmanta/inmanta-core that referenced this pull request Jul 21, 2022
Bumps [pip](https://github.com/pypa/pip) from 22.1.2 to 22.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>22.2 (2022-07-21)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>Remove the <code>html5lib</code> deprecated feature flag. (<code>[#10825](pypa/pip#10825) &lt;https://github.com/pypa/pip/issues/10825&gt;</code>_)</li>
<li>Remove <code>--use-deprecated=backtrack-on-build-failures</code>. (<code>[#11241](pypa/pip#11241) &lt;https://github.com/pypa/pip/issues/11241&gt;</code>_)</li>
</ul>
<h2>Features</h2>
<ul>
<li>
<p>Add support to use <code>truststore &lt;https://pypi.org/project/truststore/&gt;</code>_ as an
alternative SSL certificate verification backend. The backend can be enabled on Python
3.10 and later by installing <code>truststore</code> into the environment, and adding the
<code>--use-feature=truststore</code> flag to various pip commands.</p>
<p><code>truststore</code> differs from the current default verification backend (provided by
<code>certifi</code>) in it uses the operating system’s trust store, which can be better
controlled and augmented to better support non-standard certificates. Depending on
feedback, pip may switch to this as the default certificate verification backend in
the future. (<code>[#11082](pypa/pip#11082) &lt;https://github.com/pypa/pip/issues/11082&gt;</code>_)</p>
</li>
<li>
<p>Add <code>--dry-run</code> option to <code>pip install</code>, to let it print what it would install but
not actually change anything in the target environment. (<code>[#11096](pypa/pip#11096) &lt;https://github.com/pypa/pip/issues/11096&gt;</code>_)</p>
</li>
<li>
<p>Record in wheel cache entries the URL of the original artifact that was downloaded
to build the cached wheels. The record is named <code>origin.json</code> and uses the PEP 610
Direct URL format. (<code>[#11137](pypa/pip#11137) &lt;https://github.com/pypa/pip/issues/11137&gt;</code>_)</p>
</li>
<li>
<p>Support <code>PEP 691 &lt;https://peps.python.org/pep-0691/&gt;</code><em>. (<code>[#11158](pypa/pip#11158) &lt;https://github.com/pypa/pip/issues/11158&gt;</code></em>)</p>
</li>
<li>
<p>pip's deprecation warnings now subclass the built-in <code>DeprecationWarning</code>, and
can be suppressed by running the Python interpreter with
<code>-W ignore::DeprecationWarning</code>. (<code>[#11225](pypa/pip#11225) &lt;https://github.com/pypa/pip/issues/11225&gt;</code>_)</p>
</li>
<li>
<p>Add <code>pip inspect</code> command to obtain the list of installed distributions and other
information about the Python environment, in JSON format. (<code>[#11245](pypa/pip#11245) &lt;https://github.com/pypa/pip/issues/11245&gt;</code>_)</p>
</li>
<li>
<p>Significantly speed up isolated environment creation, by using the same
sources for pip instead of creating a standalone installation for each
environment. (<code>[#11257](pypa/pip#11257) &lt;https://github.com/pypa/pip/issues/11257&gt;</code>_)</p>
</li>
<li>
<p>Add an experimental <code>--report</code> option to the install command to generate a JSON report
of what was installed. In combination with <code>--dry-run</code> and <code>--ignore-installed</code> it
can be used to resolve the requirements. (<code>[#53](pypa/pip#53) &lt;https://github.com/pypa/pip/issues/53&gt;</code>_)</p>
</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Fix <code>pip install --pre</code> for packages with pre-release build dependencies defined
both in <code>pyproject.toml</code>'s <code>build-system.requires</code> and <code>setup.py</code>'s
<code>setup_requires</code>. (<code>[#10222](pypa/pip#10222) &lt;https://github.com/pypa/pip/issues/10222&gt;</code>_)</li>
<li>When pip rewrites the shebang line in a script during wheel installation,
update the hash and size in the corresponding <code>RECORD</code> file entry. (<code>[#10744](pypa/pip#10744) &lt;https://github.com/pypa/pip/issues/10744&gt;</code>_)</li>
<li>Do not consider a <code>.dist-info</code> directory found inside a wheel-like zip file
as metadata for an installed distribution. A package in a wheel is (by</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/8e7e76e60f4e115ea1201bee2f176377a718fce1"><code>8e7e76e</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/b6f6a94e36f10a4535ea5bbdc6b351f62003eede"><code>b6f6a94</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/790725aca3f60c745e33827a6079d9600da373d8"><code>790725a</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11274">#11274</a> from sbidoul/install-report-note-sbi</li>
<li><a href="https://github.com/pypa/pip/commit/d4b9e187aa7cc5ab14b2339f6171f7f2ea6504e9"><code>d4b9e18</code></a> Add clarifications to the installation report documentation</li>
<li><a href="https://github.com/pypa/pip/commit/b1a01ef762a78af1194958a1c874015eaf81fd04"><code>b1a01ef</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11265">#11265</a> from finnagin/main</li>
<li><a href="https://github.com/pypa/pip/commit/48bcb0a4ccd30a9d00e58fe58827772e307a7e39"><code>48bcb0a</code></a> reformat to pass pre-commit check</li>
<li><a href="https://github.com/pypa/pip/commit/a7c1fe3bff5655393018c53b448b669b3525515b"><code>a7c1fe3</code></a> Remove utc fixture from tests</li>
<li><a href="https://github.com/pypa/pip/commit/0c574f72905185d62bcca741c813df9bae1d9282"><code>0c574f7</code></a> Remove time import</li>
<li><a href="https://github.com/pypa/pip/commit/246fef19149eea893f1cf3efd53f9b17c94c952f"><code>246fef1</code></a> Remove utc fixture</li>
<li><a href="https://github.com/pypa/pip/commit/c9cb7f4629bdd8c61b792feff6dacb1d2e848d57"><code>c9cb7f4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/11270">#11270</a> from uranusjr/upgrade-pre-commit-hooks</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/22.1.2...22.2">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=22.1.2&new-version=22.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jul 31, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: enhancement Improvements to functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants