Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use LZMA for distribution bundles #567

Closed
stephanosio opened this issue Oct 4, 2022 · 3 comments · Fixed by #570
Closed

Use LZMA for distribution bundles #567

stephanosio opened this issue Oct 4, 2022 · 3 comments · Fixed by #570
Assignees
@stephanosio
Labels
area: CI Issues related to Continuous Integration area: Distribution Bundle Issues related to Distribution Bundle (tarball) enhancement
Milestone
0.16.0

Comments

@stephanosio
Copy link
Member

From #287 (comment):

GitHub does not allow uploading release assets larger than 2GB and we will not be able to upload the "all-in-one" distribution bundles to the GitHub releases with the picolibc also bundled in the toolchains.

We can resolve this in one of the following ways:

  1. Upload the "all-in-one" distribution bundles to a different file hosting service (e.g. S3).
    • While this would be the simplest solution, it will incur a lot of extra cost in terms of outbound traffic.
    • With the size of the distribution bundles and the number of downloads, this can be in the range of thousands of dollars per month.
  2. Use a better compression algorithm
    • Zephyr SDK currently uses GZip (tar.gz) for Linux and macOS, and DEFLATE (zip) for Windows.
    • GZip and DEFLATE are not exactly known for having the best data compression ratio.
    • Using LZMA should give much better compression ratio (a quick test has shown around 1.5x for the "all-in-one" distribution bundle with xz-utils).
    • Use XZ (tar.xz) for Linux and macOS, and 7z/LZMA (7z) for Windows. This should keep the "all-in-one" distribution bundle sizes to below 2GB.
    • LZMA requires more processing power and takes longer to compress and decompress, but this is not so much as to prevent its adoption (a quick test has shown that decompressing "all-in-one" distribution bundle tar.gz took approx. 1 minute whereas tar.xz took approx. 2.5 minutes on my machine, which is not that bad).
    • Of course, even after switching to LZMA, future changes may increase the distribution bundle sizes to more than 2GB, in which case we will have to evaluate the multi-volume archive approach described below.
  3. Use multi-volume archives
    • As its name states. Split single compressed archive into multiple volumes/parts, which can be combined together after download and decompressed.
@stephanosio stephanosio added area: CI Issues related to Continuous Integration area: Distribution Bundle Issues related to Distribution Bundle (tarball) labels Oct 4, 2022
@stephanosio stephanosio added this to the 0.16.0 milestone Oct 4, 2022
@stephanosio stephanosio self-assigned this Oct 4, 2022
stephanosio added a commit to zephyrproject-rtos/sdk-ng-testing that referenced this issue Oct 4, 2022
@stephanosio
ci: Use tar.xz format for Linux and macOS build artifacts
a4cdea8 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

For more details, refer to the issue zephyrproject-rtos/sdk-ng#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit to zephyrproject-rtos/sdk-ng-testing that referenced this issue Oct 4, 2022
@stephanosio
ci: Use tar.xz format for Linux and macOS build artifacts
fda08a9 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue zephyrproject-rtos/sdk-ng#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit to zephyrproject-rtos/sdk-ng-testing that referenced this issue Oct 4, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
311ce7c 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue zephyrproject-rtos/sdk-ng#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
@keith-packard
Copy link
Collaborator

Switching to LZMA gives us a temporary reprieve and doesn't change workflow for users, so it's a pretty simple choice to make. Moving forward, it's pretty clear we'll need a solution that breaks the image into pieces at some point, that could also allow inclusion of debug symbols if the pieces are small enough. I think that would require writing a tool to help developers manage the SDK installation?

@stephanosio
Copy link
Member Author

I think that would require writing a tool to help developers manage the SDK installation?

zephyrproject-rtos/zephyr#37255 is basically where we want to be, eventually.

stephanosio added a commit to stephanosio/zephyr-sdk-ng that referenced this issue Oct 4, 2022
@stephanosio
ci: Use tar.xz format for Linux and macOS build artifacts
d373fb6 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue zephyrproject-rtos#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit to stephanosio/zephyr-sdk-ng that referenced this issue Oct 4, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
64786ef 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue zephyrproject-rtos#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
@stephanosio stephanosio mentioned this issue Oct 4, 2022
Merged
stephanosio added a commit to stephanosio/zephyr-sdk-ng that referenced this issue Oct 5, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
88b0bfa 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue zephyrproject-rtos#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit to zephyrproject-rtos/sdk-ng-testing that referenced this issue Oct 5, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
eb628cd 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue zephyrproject-rtos/sdk-ng#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit to zephyrproject-rtos/sdk-ng-testing that referenced this issue Oct 6, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
8819f8b 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue zephyrproject-rtos/sdk-ng#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit to stephanosio/zephyr-sdk-ng that referenced this issue Oct 6, 2022
@stephanosio
ci: Use tar.xz format for Linux and macOS build artifacts
1624d05 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue zephyrproject-rtos#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit to stephanosio/zephyr-sdk-ng that referenced this issue Oct 6, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
647a506 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue zephyrproject-rtos#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit that referenced this issue Oct 11, 2022
@stephanosio
ci: Use tar.xz format for Linux and macOS build artifacts
9226f49 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit that referenced this issue Oct 11, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
2433759 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 19, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
2ebbca0 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 19, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
53e18dd 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit to keith-packard/sdk-ng that referenced this issue Oct 20, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
abe9a90 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue zephyrproject-rtos#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit to keith-packard/sdk-ng that referenced this issue Oct 20, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
eb934dc 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue zephyrproject-rtos#567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 21, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
3794fa0 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 21, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
660c9da 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 22, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
c354967 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 22, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
0b9ff8d 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 22, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
330834f 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 22, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
33d90e9 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 24, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
8b04418 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 24, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
05cf202 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 25, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
94e786f 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Oct 25, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
1926b37 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Nov 3, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
f784e52 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Nov 3, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
b403bf7 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Nov 24, 2022
@stephanosio @keith-packard
ci: Use tar.xz format for Linux and macOS build artifacts
62ad14a 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
keith-packard pushed a commit that referenced this issue Nov 24, 2022
@stephanosio @keith-packard
ci: Use 7z format for Windows build artifacts
de414e7 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit that referenced this issue Nov 25, 2022
@stephanosio
ci: Use tar.xz format for Linux and macOS build artifacts
f95874d 
This commit updates the CI workflow to output .tar.xz archives for the
Linux and macOS build artifacts.

The XZ format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

Note that the `-T0` XZ option is used to perform multi-threaded
compression in spite of reduced compression ratio; otherwise,
compressing the distribution bundle may take tens of minutes to
complete.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
stephanosio added a commit that referenced this issue Nov 25, 2022
@stephanosio
ci: Use 7z format for Windows build artifacts
5aa5274 
This commit updates the CI workflow to output .7z archives for the
Windows build artifacts.

The 7z format uses the LZMA compression algorithm, which offers
significantly improved data compression ratio.

The `-l` option is used such that symbolic links are stored as copies
of the linked files because Windows requires administrator privileges
to create symbolic links.

Note that 7z performs multi-threaded compression by default.

For more details, refer to the issue #567.

Signed-off-by: Stephanos Ioannidis <root@stephanos.io>
@stephanosio
Copy link
Member Author

Merged to main as part of #602

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stephanosio stephanosio

Labels
area: CI Issues related to Continuous Integration area: Distribution Bundle Issues related to Distribution Bundle (tarball) enhancement
Projects
None yet
Milestone
0.16.0
Development

Successfully merging a pull request may close this issue.

ci: Use LZMA compression for build artifacts stephanosio/zephyr-sdk-ng
2 participants
@keith-packard @stephanosio