Skip to content

Commit

Permalink
Documentation fixes and fixes from testing 1.1.1 (#125)
Browse files Browse the repository at this point in the history
* minor fixes for gnu make builds
* updates to documentation
* update version numbers in docs and in headers
  • Loading branch information
markcmiller86 committed Aug 30, 2023
1 parent 4ae3042 commit 51d2218
Show file tree
Hide file tree
Showing 9 changed files with 294 additions and 344 deletions.
16 changes: 13 additions & 3 deletions config.make
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ ifeq ($(PWD_BASE),src)
H5Z_ZFP_BASE := .
else ifeq ($(PWD_BASE),test)
H5Z_ZFP_BASE := ../src
else ifeq ($(PWD_BASE),H5Z-ZFP)
else ifeq ($(findstring H5Z-ZFP, $(PWD_BASE)), H5Z-ZFP)
H5Z_ZFP_BASE := ./src
endif

H5Z_ZFP_PLUGIN := $(H5Z_ZFP_BASE)/plugin
H5Z_ZFP_VERSINFO := $(shell grep '^\#define H5Z_FILTER_ZFP_VERSION_[MP]' $(H5Z_ZFP_BASE)/H5Zzfp_plugin.h | cut -d' ' -f3 | tr '\n' '.' | cut -d'.' -f-3 2>/dev/null)
H5Z_ZFP_VERSINFO := $(shell grep '^\#define H5Z_FILTER_ZFP_VERSION_[MP]' $(H5Z_ZFP_BASE)/H5Zzfp_version.h | cut -d' ' -f3 | tr '\n' '.' | cut -d'.' -f-3 2>/dev/null)
ZFP_HAS_REVERSIBLE :=
ifneq ($(ZFP_HOME),)
ZFP_HAS_REVERSIBLE := $(shell grep zfp_stream_set_reversible $(ZFP_HOME)/include/zfp.h 2>/dev/null)
Expand Down Expand Up @@ -67,6 +67,7 @@ HAS_ICC := $(shell basename $$(which icc 2>/dev/null) 2>/dev/null)
HAS_PGCC := $(shell basename $$(which pgcc 2>/dev/null) 2>/dev/null)
HAS_XLCR := $(shell basename $$(which xlc_r 2>/dev/null) 2>/dev/null)
HAS_BGXLCR := $(shell basename $$(which bgxlc_r 2>/dev/null) 2>/dev/null)
HAS_CC := $(shell basename $$(which cc 2>/dev/null) 2>/dev/null)

# Common Fortran compilers
HAS_GFORTRAN := $(shell basename $$(which gfortran 2>/dev/null) 2>/dev/null)
Expand All @@ -81,6 +82,8 @@ ifeq ($(CC),)
CC = $(HAS_CLANG)
else ifneq ($(strip $(HAS_GCC)),)
CC = $(HAS_GCC)
else ifneq ($(strip $(HAS_CC)),)
CC = $(HAS_CC)
endif
else ifneq ($(findstring ppc, $(PROCESSOR),),)
ifneq ($(strip $(HAS_BGXLCR)),)
Expand All @@ -93,6 +96,8 @@ ifeq ($(CC),)
else
ifneq ($(strip $(HAS_GCC)),)
CC = $(HAS_GCC)
else ifneq ($(strip $(HAS_CLANG)),)
CC = $(HAS_CLANG)
else ifneq ($(strip $(HAS_ICC)),)
CC = $(HAS_ICC)
else ifneq ($(strip $(HAS_PGCC)),)
Expand All @@ -117,7 +122,7 @@ ifneq ($(findstring gcc, $(CC)),)
else ifneq ($(findstring clang, $(CC)),)
SOEXT ?= dylib
SHFLAG ?= -dynamiclib
PREPATH = -L
PREPATH = -Wl,-rpath,
else ifneq ($(findstring icc, $(CC)),)
CFLAGS += -fpic
SOEXT ?= so
Expand All @@ -128,6 +133,11 @@ else ifneq ($(findstring pgcc, $(CC)),)
SOEXT ?= so
SHFLAG ?= -shared
PREPATH = -Wl,-rpath,
else ifneq ($(findstring cc, $(CC)),)
CFLAGS += -fPIC
SOEXT ?= so
SHFLAG ?= -shared
PREPATH = -Wl,-rpath,
else ifneq ($(findstring xlc_r, $(CC)),)
CFLAGS += -qpic
SOEXT ?= so
Expand Down
4 changes: 4 additions & 0 deletions docs/cd_vals.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
H5Z-ZFP and the HDF5 filter's cd_values
=======================================

.. note::
The details described here are likely relevant only to *developers* of the H5Z-ZFP_ filter.
If you just want to *use* the filter, you can ignore this material.

The HDF5_ library uses an array of values, named ``cd_values`` in formal arguments documenting various API functions, for managing *auxiliary data* for a filter.
Instances of this ``cd_values`` array are used in two subtly different ways within HDF5.

Expand Down
95 changes: 0 additions & 95 deletions docs/cmake.rst

This file was deleted.

3 changes: 2 additions & 1 deletion docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
.. _HDF5: https://docs.hdfgroup.org/hdf5/develop/
.. _H5Z-ZFP: https://github.com/LLNL/H5Z-ZFP
.. _Spack: https://spack.io
.. _CMake: https://cmake.org/cmake/help/latest/
"""

# The version info for the project you're documenting, acts as replacement for
Expand All @@ -60,7 +61,7 @@
# The short X.Y version.
version = '1.1'
# The full version, including alpha/beta/rc tags.
release = '1.1.0'
release = '1.1.1'

# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
Expand Down
98 changes: 44 additions & 54 deletions docs/h5repack.rst
Original file line number Diff line number Diff line change
@@ -1,76 +1,69 @@
==================================
Using H5Z-ZFP Plugin with H5Repack
==================================
A convenient way to use and play with the ZFP_ filter is a *plugin* with
the HDF5_ `h5repack <https://portal.hdfgroup.org/display/HDF5/h5repack>`__ (**note**: the preceding link to ``h5repack`` may not render correctly from Safari)
utility using the ``-f`` filter argument to apply ZFP to existing data in a file.
A convenient way to use and play with the ZFP_ filter is as a *plugin* with the HDF5_ `h5repack <https://docs.hdfgroup.org/hdf5/develop/_view_tools_edit.html#secViewToolsEditApply>`__ utility using the ``-f`` command-line argument to apply ZFP to existing data in a file.

-----------------
Patching h5repack
-----------------
Some versions of HDF5_'s ``h5repack`` utility contain a bug that prevents
proper parsing of the ``-f`` argument's option. In order to use ``h5repack``
with ``-f`` argument as described here, you need to apply the patch from
`h5repack_parse.patch <https://github.com/LLNL/H5Z-ZFP/blob/master/test/h5repack_parse.patch>`_.
To do so, after you've downloaded and untar'd HDF5_ but before you've built
it, do something like the following using HDF5-1.8.14 as an example::

gunzip < hdf5-1.8.14.tar.gz | tar xvf -
cd hdf5-1.8.14
patch ./tools/h5repack/h5repack_parse.c <path-to-H5Z-ZFP-test-dir>/h5repack_parse.patch
.. warning::

Versions of HDF5_'s ``h5repack`` utility prior to 1.10.4 contain a bug that prevents proper parsing of the ``-f`` argument's option.
In order to use ``h5repack`` with ``-f`` argument as described here, you need to apply the patch from `h5repack_parse.patch <https://github.com/LLNL/H5Z-ZFP/blob/master/test/h5repack_parse.patch>`_.
To do so, after you've downloaded and untar'd HDF5_ but before you've built it, do something like the following using HDF5-1.8.14 as an example::

gunzip < hdf5-1.8.14.tar.gz | tar xvf -
cd hdf5-1.8.14
patch ./tools/h5repack/h5repack_parse.c <path-to-H5Z-ZFP-test-dir>/h5repack_parse.patch

-------------------------------------
Constructing an HDF5_ cd_values array
-------------------------------------
HDF5_'s ``h5repack`` utility uses only the *generic* interface to HDF5_ filters.
Another challenge in using ``h5repack`` as described here is constructing the set
``unsigned int cd_values`` as is used in
`H5Pset_filter() <https://docs.hdfgroup.org/hdf5/develop/group___o_c_p_l.html#ga191c567ee50b2063979cdef156a768c5>`__
required by the *generic* HDF5_ filter interface, especially because
of the type-punning (doubles as unsigned int) involved.

**Note:** Querying an existing dataset using ``h5dump`` or ``h5ls`` to obtain
the *cd_values* stored with a ZFP_ compressed dataset
**will not provide the correct cd_values**. This is because the *cd_values*
stored in the file are different from those used in the *generic* interface
to invoke the ZFP_ filter.

To facilitate constructing a valid ``-f`` argument to ``h5repack``, we have
created a utility program, ``print_h5repack_farg``, which is presently in the
``test`` directory and is built when tests are built.

You can use the ``print_h5repack_farg`` utility to read a command-line
consisting of ZFP_ filter parameters you wish to use and it will output
part of the command-line needed for the ``-f`` argument to ``h5repack``.
Another challenge in using ``h5repack`` as described here is constructing the set ``unsigned int cd_values`` as is used in `H5Pset_filter() <https://docs.hdfgroup.org/hdf5/develop/group___o_c_p_l.html#ga191c567ee50b2063979cdef156a768c5>`__ required by the *generic* HDF5_ filter interface, especially because of the type-punning (doubles as unsigned int) which may be involved.

.. note::

Querying an existing dataset using ``h5dump`` or ``h5ls`` to obtain the ``cd_values`` *stored* with a ZFP_ compressed dataset will not provide the correct ``cd_values`` needed to invoke the filter.
This is because the ``cd_values`` stored in the file are different from those needed in the *generic* interface to invoke the ZFP_ filter.

To facilitate constructing a valid ``-f`` argument to ``h5repack``, there is a utility program, ``print_h5repack_farg``, which is presently in the ``test`` directory and is built when tests are built.
This program was originally written simply to facilitate testing of H5Z-ZFP filter.
It should eventually be made a first class *tool* installed with H5Z-ZFP.
However, it is presently made only as part of *testing* H5Z-ZFP.

The ``print_h5repack_farg`` utility can be used to read a command-line consisting of ZFP_ filter parameters you wish to use and output part of the command-line needed for the ``-f`` argument to ``h5repack``.

--------
Examples
--------

In the examples below, we use ``h5repack`` with the example data file,
``mesh.h5`` in the tests directory.
.. note::

The examples below assume H5Z-ZFP has been configured to run tests by either a preceding ``make check`` (if using a vanilla ``gmake`` build) or ``make test`` (if using a ``cmake`` build).

In the examples below, we use ``h5repack`` with the example data file, ``mesh.h5`` in the tests directory.

To use ZFP_ filter in *rate* mode with a rate of ``4.5`` bits per value,
first, use the ``print_h5repack_farg``::
To use ZFP_ filter in *rate* mode with a rate of ``4.5`` bits per value, first, use the ``print_h5repack_farg``::

% ./print_h5repack_farg zfpmode=1 rate=4.5
Print cdvals for set of ZFP compression paramaters...
zfpmode=1 set zfp mode (1=rate,2=prec,3=acc,4=expert)
zfpmode=1 set zfp mode (1=rate,2=prec,3=acc,4=expert,5=rev)
rate=4.5 set rate for rate mode of filter
acc=0 set accuracy for accuracy mode of filter
prec=11 set precision for precision mode of zfp filter
prec=0 set precision for precision mode of zfp filter
minbits=0 set minbits for expert mode of zfp filter
maxbits=4171 set maxbits for expert mode of zfp filter
maxprec=64 set maxprec for expert mode of zfp filter
minexp=-1074 set minexp for expert mode of zfp filter
maxbits=0 set maxbits for expert mode of zfp filter
maxprec=0 set maxprec for expert mode of zfp filter
minexp=0 set minexp for expert mode of zfp filter
help=0 this help message

h5repack -f argument...
-f UD=32013,0,4,1,0,0,1074921472

Next, cut-n-paste the ``-f UD=32013,0,4,1,0,0,1074921472`` in a command
to ``h5repack`` like so::
Next, cut-n-paste the ``-f UD=32013,0,4,1,0,0,1074921472`` in a command to ``h5repack`` like so::

env LD_LIBRARY_PATH=<path-to-dir-with-libhdf5.so>:$(LD_LIBRARY_PATH) \
HDF5_PLUGIN_PATH=<path-to-dir-with-libh5zzfp.so> \
Expand All @@ -80,30 +73,27 @@ to ``h5repack`` like so::
-l VelocityX_2D:CHUNK=21x31 \
mesh.h5 mesh_repack.h5

where the ``-l`` arguments indicate the dataset(s) to be re-packed as well
as their (new) chunking.
where the ``-l`` arguments indicate the dataset(s) to be re-packed as well as their (new) chunking.

To use ZFP_ filter in *accuracy* mode with an accuracy of ``0.075``,
first, use the ``print_h5repack_farg``::
To use ZFP_ filter in *accuracy* mode with an accuracy of ``0.075``, first, use the ``print_h5repack_farg``::

% ./print_h5repack_farg zfpmode=3 acc=0.075
Print cdvals for set of ZFP compression paramaters...
zfpmode=3 set zfp mode (1=rate,2=prec,3=acc,4=expert)
rate=4 set rate for rate mode of filter
zfpmode=3 set zfp mode (1=rate,2=prec,3=acc,4=expert,5=rev)
rate=3.5 set rate for rate mode of filter
acc=0.075 set accuracy for accuracy mode of filter
prec=11 set precision for precision mode of zfp filter
prec=0 set precision for precision mode of zfp filter
minbits=0 set minbits for expert mode of zfp filter
maxbits=4171 set maxbits for expert mode of zfp filter
maxprec=64 set maxprec for expert mode of zfp filter
minexp=-1074 set minexp for expert mode of zfp filter
maxbits=0 set maxbits for expert mode of zfp filter
maxprec=0 set maxprec for expert mode of zfp filter
minexp=0 set minexp for expert mode of zfp filter
help=0 this help message

h5repack -f argument...
-f UD=32013,0,4,3,0,858993459,1068708659

Next, cut-n-paste the ``-f UD=32013,0,4,3,0,858993459,1068708659`` in a command
to ``h5repack`` like so::
Next, cut-n-paste the ``-f UD=32013,0,4,3,0,858993459,1068708659`` in a command to ``h5repack`` like so::

env LD_LIBRARY_PATH=<path-to-dir-with-libhdf5.so>:$(LD_LIBRARY_PATH) \
HDF5_PLUGIN_PATH=<path-to-dir-with-libh5zzfp.so> \
Expand Down
1 change: 0 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,3 @@ Contents:
h5repack
endian_issues
tests
cmake
Loading

0 comments on commit 51d2218

Please sign in to comment.