Skip to content

Commit

Permalink
Merge pull request #137 from t20100/sphinx
Browse files Browse the repository at this point in the history 
LGTM
  • Loading branch information
@vasole
vasole authored Jun 25, 2021
2 parents f3aacc5 + 3de2f3c commit f0ac499
Show file tree
Hide file tree
Showing 13 changed files with 682 additions and 397 deletions.
269 changes: 10 additions & 259 deletions README.rst
View file
Original file line number Diff line number Diff line change
@@ -1,23 +1,9 @@
hdf5plugin
==========

This module provides HDF5 compression filters (namely: blosc, bitshuffle, lz4, FCIDECOMP, ZFP, zstd) and registers them to the HDF5 library used by `h5py <https://www.h5py.org>`_.
*hdf5plugin* provides HDF5 compression filters (namely: blosc, bitshuffle, lz4, FCIDECOMP, ZFP, zstd) and makes them usable from `h5py <https://www.h5py.org>`_.

* Supported operating systems: Linux, Windows, macOS.
* Supported versions of Python: >= 3.4

`hdf5plugin` provides a generic way to enable the use of the provided HDF5 compression filters with `h5py` that can be installed via `pip` or `conda`.

Alternatives to install HDF5 compression filters are: system-wide installation on Linux or other conda packages: `blosc-hdf5-plugin <https://anaconda.org/conda-forge/blosc-hdf5-plugin>`_, `hdf5-lz4 <https://anaconda.org/nsls2forge/hdf5-lz4>`_.

The HDF5 plugin sources were obtained from:

* LZ4 plugin (v0.1.0) and lz4 (v1.3.0, tag r122): https://github.com/nexusformat/HDF5-External-Filter-Plugins, https://github.com/lz4/lz4
* bitshuffle plugin (0.3.5): https://github.com/kiyo-masui/bitshuffle
* hdf5-blosc plugin (v1.0.0), c-blosc (v1.20.1) and snappy (v1.1.1): https://github.com/Blosc/hdf5-blosc, https://github.com/Blosc/c-blosc and https://github.com/Blosc/c-blosc/tree/v1.17.0/internal-complibs/snappy-1.1.1
* FCIDECOMP plugin (v1.0.2) and CharLS (branch 1.x-master SHA1 ID:25160a42fb62e71e4b0ce081f5cb3f8bb73938b5): ftp://ftp.eumetsat.int/pub/OPS/out/test-data/Test-data-for-External-Users/MTG_FCI_Test-Data/FCI_Decompression_Software_V1.0.2/ and https://github.com/team-charls/charls.git
* HDF5-ZFP plugin (v1.0.1) and zfp (v0.5.5): https://github.com/LLNL/H5Z-ZFP and https://github.com/LLNL/zfp
* HDF5Plugin-Zstandard (commit d5afdb5) and zstd (v1.4.5): https://github.com/aparamon/HDF5Plugin-Zstandard and https://github.com/Blosc/c-blosc/tree/v1.20.1/internal-complibs/zstd-1.4.5
See `documentation <http://www.silx.org/doc/hdf5plugin/latest/>`_.

Installation
------------
Expand All @@ -36,254 +22,19 @@ To install from source and recompile the HDF5 plugins, run::

Installing from source can achieve better performances by enabling AVX2 and OpenMP if available.

Documentation
-------------

To use it, just use ``import hdf5plugin`` and supported compression filters are available from `h5py <https://www.h5py.org>`_.

Sample code:

.. code-block:: python
import numpy
import h5py
import hdf5plugin
# Compression
f = h5py.File('test.h5', 'w')
f.create_dataset('data', data=numpy.arange(100), **hdf5plugin.LZ4())
f.close()
# Decompression
f = h5py.File('test.h5', 'r')
data = f['data'][()]
f.close()
``hdf5plugin`` provides:

* Compression option helper classes to prepare arguments to provide to ``h5py.Group.create_dataset``:

- `Bitshuffle(nelems=0, lz4=True)`_
- `Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE)`_
- `FciDecomp()`_
- `LZ4(nbytes=0)`_
- `Zfp()`_


* The HDF5 filter ID of embedded plugins:

- ``BLOSC_ID``
- ``BSHUF_ID``
- ``FCIDECOMP_ID``
- ``LZ4_ID``
- ``ZFP_ID``
- ``ZSTD_ID``

* ``FILTERS``: A dictionary mapping provided filters to their ID
* ``PLUGINS_PATH``: The directory where the provided filters library are stored.

Bitshuffle(nelems=0, lz4=True)
******************************

This class takes the following arguments and returns the compression options to feed into ``h5py.Group.create_dataset`` for using the bitshuffle filter:

* **nelems** the number of elements per block, needs to be divisible by eight (default is 0, about 8kB per block)
* **lz4** if True the elements get compressed using lz4 (default is True)

It can be passed as keyword arguments.

Sample code:

.. code-block:: python
f = h5py.File('test.h5', 'w')
f.create_dataset('bitshuffle_with_lz4', data=numpy.arange(100),
**hdf5plugin.Bitshuffle(nelems=0, lz4=True))
f.close()
Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE)
*********************************************

This class takes the following arguments and returns the compression options to feed into ``h5py.Group.create_dataset`` for using the blosc filter:

* **cname** the compression algorithm, one of:

* 'blosclz'
* 'lz4' (default)
* 'lz4hc'
* 'snappy' (optional, requires C++11)
* 'zlib'
* 'zstd'

* **clevel** the compression level, from 0 to 9 (default is 5)
* **shuffle** the shuffling mode, in:

* `Blosc.NOSHUFFLE` (0): No shuffle
* `Blosc.SHUFFLE` (1): byte-wise shuffle (default)
* `Blosc.BITSHUFFLE` (2): bit-wise shuffle

It can be passed as keyword arguments.

Sample code:

.. code-block:: python
f = h5py.File('test.h5', 'w')
f.create_dataset('blosc_byte_shuffle_blosclz', data=numpy.arange(100),
**hdf5plugin.Blosc(cname='blosclz', clevel=9, shuffle=hdf5plugin.Blosc.SHUFFLE))
f.close()
FciDecomp()
***********

This class returns the compression options to feed into ``h5py.Group.create_dataset`` for using the FciDecomp filter:

It can be passed as keyword arguments.

Sample code:

.. code-block:: python
f = h5py.File('test.h5', 'w')
f.create_dataset('fcidecomp', data=numpy.arange(100),
**hdf5plugin.FciDecomp())
f.close()
LZ4(nbytes=0)
*************

This class takes the number of bytes per block as argument and returns the compression options to feed into ``h5py.Group.create_dataset`` for using the lz4 filter:

* **nbytes** number of bytes per block needs to be in the range of 0 < nbytes < 2113929216 (1,9GB).
The default value is 0 (for 1GB).

It can be passed as keyword arguments.
For more details, see the `installation documentation <http://www.silx.org/doc/hdf5plugin/latest/install.html>`_.

Sample code:
How-to use
----------

.. code-block:: python
f = h5py.File('test.h5', 'w')
f.create_dataset('lz4', data=numpy.arange(100),
**hdf5plugin.LZ4(nbytes=0))
f.close()
Zfp()
*****

This class returns the compression options to feed into ``h5py.Group.create_dataset`` for using the zfp filter:

It can be passed as keyword arguments.

Sample code:

.. code-block:: python
f = h5py.File('test.h5', 'w')
f.create_dataset('zfp', data=numpy.random.random(100),
**hdf5plugin.Zfp())
f.close()
The zfp filter compression mode is defined by the provided arguments.
The following compression modes are supported:

- **Fixed-rate** mode:
For details, see `zfp fixed-rate mode <https://zfp.readthedocs.io/en/latest/modes.html#fixed-rate-mode>`_.

.. code-block:: python
f.create_dataset('zfp_fixed_rate', data=numpy.random.random(100),
**hdf5plugin.Zfp(rate=10.0))
- **Fixed-precision** mode:
For details, see `zfp fixed-precision mode <https://zfp.readthedocs.io/en/latest/modes.html#fixed-precision-mode>`_.

.. code-block:: python
f.create_dataset('zfp_fixed_precision', data=numpy.random.random(100),
**hdf5plugin.Zfp(precision=10))
- **Fixed-accuracy** mode:
For details, see `zfp fixed-accuracy mode <https://zfp.readthedocs.io/en/latest/modes.html#fixed-accuracy-mode>`_.

.. code-block:: python
f.create_dataset('zfp_fixed_accuracy', data=numpy.random.random(100),
**hdf5plugin.Zfp(accuracy=0.001))
- **Reversible** (i.e., lossless) mode:
For details, see `zfp reversible mode <https://zfp.readthedocs.io/en/latest/modes.html#reversible-mode>`_.

.. code-block:: python
f.create_dataset('zfp_reversible', data=numpy.random.random(100),
**hdf5plugin.Zfp(reversible=True))
- **Expert** mode:
For details, see `zfp expert mode <https://zfp.readthedocs.io/en/latest/modes.html#expert-mode>`_.

.. code-block:: python
f.create_dataset('zfp_expert', data=numpy.random.random(100),
**hdf5plugin.Zfp(minbits=1, maxbits=16657, maxprec=64, minexp=-1074))
Zstd()
******

This class returns the compression options to feed into ``h5py.Group.create_dataset`` for using the Zstd filter:

It can be passed as keyword arguments.

Sample code:

.. code-block:: python
f = h5py.File('test.h5', 'w')
f.create_dataset('zstd', data=numpy.arange(100),
**hdf5plugin.Zstd())
f.close()
Dependencies
------------

* `h5py <https://www.h5py.org>`_

Testing
-------

To run self-contained tests, from Python:

.. code-block:: python
import hdf5plugin.test
hdf5plugin.test.run_tests()
Or, from the command line::

python -m hdf5plugin.test

To also run tests relying on actual HDF5 files, run from the source directory::

python test/test.py
To use it, just use ``import hdf5plugin`` and supported compression filters are available from `h5py <https://www.h5py.org>`_.

This tests the installed version of `hdf5plugin`.
For details, see `Usage documentation <http://www.silx.org/doc/hdf5plugin/latest/usage.html>`_.

License
-------

The source code of *hdf5plugin* itself is licensed under the MIT license.
Use it at your own risk.
See `LICENSE <https://github.com/silx-kit/hdf5plugin/blob/master/LICENSE>`_

The source code of the embedded HDF5 filter plugin libraries is licensed under different open-source licenses.
Please read the different licenses:

* bitshuffle: See `src/bitshuffle/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/master/src/bitshuffle/LICENSE>`_
* blosc: See `src/hdf5-blosc/LICENSES/ <https://github.com/silx-kit/hdf5plugin/blob/master/src/hdf5-blosc/LICENSES/>`_ and `src/c-blosc/LICENSES/ <https://github.com/silx-kit/hdf5plugin/blob/master/src/c-blosc/LICENSES/>`_
* lz4: See `src/LZ4/COPYING <https://github.com/silx-kit/hdf5plugin/blob/master/src/LZ4/COPYING>`_ and `src/lz4-r122/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/master/src/lz4-r122/LICENSE>`_
* FCIDECOMP: See `src/fcidecomp/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/master/src/fcidecomp/LICENSE.txt>`_ and `src/charls/src/License.txt <https://github.com/silx-kit/hdf5plugin/blob/master/src/charls/License.txt>`_
* zfp: See `src/H5Z-ZFP/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/master/src/H5Z-ZFP/LICENSE>`_ and `src/zfp/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/master/src/zfp/LICENSE>`_
* zstd: See `src/HDF5Plugin-Zstandard/LICENSE`
The source code of *hdf5plugin* itself is licensed under the `MIT license <LICENSE>`_.

The HDF5 v1.10.5 headers (and Windows .lib file) used to build the filters are stored for convenience in the repository. The license is available here: `src/hdf5/COPYING <https://github.com/silx-kit/hdf5plugin/blob/master/src/hdf5/COPYING>`_.
Embedded HDF5 compression filters are licensed under different open-source licenses:
see the `license documentation <http://www.silx.org/doc/hdf5plugin/latest/information.html#license>`_.
20 changes: 20 additions & 0 deletions doc/Makefile
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
#

# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = _build

# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

.PHONY: help Makefile

# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
5 changes: 5 additions & 0 deletions doc/changelog.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
===========
Changelog
===========

.. include:: ../CHANGELOG.rst
Loading

0 comments on commit f0ac499

Please sign in to comment.