Skip to content

Commit

Permalink
Add docs for using SquashFS on targets w/o zstd
Browse files Browse the repository at this point in the history
  • Loading branch information
siboehm committed May 17, 2021
1 parent 249bada commit d34dfc9
Show file tree
Hide file tree
Showing 2 changed files with 18 additions and 8 deletions.
11 changes: 8 additions & 3 deletions conda_pack/formats.py
Original file line number Diff line number Diff line change
Expand Up @@ -402,13 +402,18 @@ def mksquashfs_from_staging(self):
cmd += ["-comp", comp_algo_str]
else:
comp_level = int(self.compress_level / 8 * 20)
comp_algo_str = "zstd level {}".format(comp_level)
comp_algo_str = "zstd (level {})".format(comp_level)
# 256KB block size instead of the default 128KB for slightly smaller archive sizes
cmd += ["-comp", "zstd", "-Xcompression-level", str(comp_level), "-b", str(256*1024)]

if self.verbose:
print("Running mksquashfs (processors: {}, compression: {})".format(
self.n_threads, comp_algo_str))
s = "Running mksquashfs with {} compression (processors: {}).".format(
comp_algo_str, self.n_threads)
if self.compress_level != 9:
s += "\nWill require kernel>=4.14 or squashfuse>=0.1.101 (compiled with zstd) " \
"for mounting.\nTo support older systems, compress with " \
"`xz` (--compress-level 9) instead."
print(s)
else:
cmd.append("-no-progress")
subprocess.check_call(cmd)
Expand Down
15 changes: 10 additions & 5 deletions docs/source/squashfs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ SquashFS
``conda-pack`` can package environments into
`SquashFS <https://en.wikipedia.org/wiki/SquashFS>`_, a compressed, read-only Linux filesystem.
These filesystems can then be mounted directly, without decompressing them first.
This allows benefiting from compressed storage, without the decompression step necessary for the
``zip`` and ``tar.gz`` formats.
This allows using packed environments immediately and without consuming disk space.

Packing
-------
Expand Down Expand Up @@ -73,7 +72,13 @@ Compression options
-------------------

Compression can be specified through ``--compress-level``.
Default is level 4.
Default is level 4, which will use ``zstd`` compression.

When selecting a compression option for packing, keep in mind the kernel or ``squashfuse`` version on the target system.
For older systems, make sure that SquashFS or ``squashfuse`` support ``zstd`` compression
(Linux kernel version ``>=4.14`` or ``squashfuse >= 0.1.101``).
If ``zstd`` isn't supported on the target, you can always compress with ``xz`` instead.


- 0: no compression
- 1-8: ``zstd`` with increasing compression level
Expand All @@ -87,10 +92,10 @@ Sometimes the unpacked environment needs to be writeable on the target machine,
more packages.
A good way to do this is to use `Union mounting <https://en.wikipedia.org/wiki/Union_mount>`_ to
add a writeable layer on top of the read-only SquashFS.

On Linux the most used option is `OverlayFS <https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html>`_.

To show how to set this up, we create three layers:
To set this up, we create three layers:

1. The SquashFS-packed conda env as a read-only lower layer
2. A writeable working directory, necessary for OverlayFS
3. A writeable upper directory, where all new and changed files will go
Expand Down

0 comments on commit d34dfc9

Please sign in to comment.