Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for using zstd-compressed nightly bundles #2442

Merged
merged 7 commits into from
Aug 30, 2024

Conversation

henrymercer
Copy link
Contributor

@henrymercer henrymercer commented Aug 23, 2024

We are currently experimenting with using bundles compressed using Zstandard (zstd) rather than gzip to potentially save a significant chunk of the time we spend extracting the bundle. This could lead to a noticeable analysis speedup, particularly for small codebases where fixed costs represent a higher proportion of the total runtime.

This PR is one of the first steps. Specifically:

  • We allow users to pass zstd bundles using the tools input to init
  • We add support for decompressing these zstd bundles. This works by letting tar auto-detect the compression method, i.e. instead of running tar xz, x for extract, z for gzip, we just run tar x.
  • We add telemetry for the compression method used and the version of tar (this informs us whether zstd should be available).
  • We download zstd bundles in PR checks to test out this functionality (and hopefully speed up the PR checks!)

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Confirm the readme has been updated if necessary.
  • Confirm the changelog has been updated if necessary.

@henrymercer henrymercer force-pushed the henrymercer/zstd-bundles branch 3 times, most recently from 4b5b9e6 to 7ce060f Compare August 28, 2024 15:25
@henrymercer henrymercer marked this pull request as ready for review August 29, 2024 17:16
@henrymercer henrymercer requested a review from a team as a code owner August 29, 2024 17:16
Copy link
Contributor

@angelapwen angelapwen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, couple of questions:

  • is there a reason we're only supporting for nightly bundles so far? (maybe I've missed the context behind that)
  • should we expose this in an experimental changenote or will we do that once we've deemed the experiment a success?

@henrymercer
Copy link
Contributor Author

henrymercer commented Aug 29, 2024

Thanks for the review!

  • is there a reason we're only supporting for nightly bundles so far? (maybe I've missed the context behind that)

Only that we haven't had a stable release with a zstd bundle yet. My thinking was that it would be easier to write and test the functionality for stable versions once we have a stable zstd bundle.

  • should we expose this in an experimental changenote or will we do that once we've deemed the experiment a success?

I was thinking no since this only applies to nightly bundles which we haven't advertised before, but I don't feel strongly about it.

@aeisenberg
Copy link
Contributor

  • should we expose this in an experimental changenote or will we do that once we've deemed the experiment a success?

I was thinking no since this only applies to nightly bundles which we haven't advertised before, but I don't feel strongly about it.

When we're confident about this being a positive improvement, we will go through a proper release channel just to make sure we put out a changelog note and update documentation.

@henrymercer henrymercer merged commit 821ab42 into main Aug 30, 2024
611 of 613 checks passed
@henrymercer henrymercer deleted the henrymercer/zstd-bundles branch August 30, 2024 10:42
@github-actions github-actions bot mentioned this pull request Sep 13, 2024
8 tasks
@dbartol dbartol mentioned this pull request Sep 13, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants