Support for Dynamically Loaded Filter plugins

[AlexHenderson]

Hello,

I'm trying to design a dynamic filter plugin for (very) sparse data. I assume no one has already done that!
Does HighFive have a method of setting a dynamic filter? It looks as though only the built-in method is available.
ref: Dynamically Loaded Filters

Thanks, Alex

[1uc]

Good morning, HighFive is likely of zero value when writing such a filter. Only after it has been written could it, if desired, be wrapped for HighFive.

To get the filter to play "nicely" with HighFive there's two choices:

1. Use the HDF5 and filter's API directly, by using .getId() on the required HighFive objects.
2. Wrap the Filters API in a style similar to what HighFive does for other properties:
   

   HighFive/include/highfive/H5PropertyList.hpp

   Lines 247 to 262 in 9cbd834

   	class MPIOCollectiveMetadata {
   	public:
   	explicit MPIOCollectiveMetadata(bool collective = true);
   	explicit MPIOCollectiveMetadata(const FileAccessProps& plist);
   	bool isCollectiveRead() const;
   	bool isCollectiveWrite() const;
   	private:
   	friend FileAccessProps;
   	void apply(hid_t plist) const;
   	bool collective_read_;
   	bool collective_write_;
   	};

   
   

   HighFive/include/highfive/bits/H5PropertyList_misc.hpp

   Lines 193 to 210 in 9cbd834

   	inline void MPIOCollectiveMetadataRead::apply(const hid_t plist) const {
   	detail::h5p_set_all_coll_metadata_ops(plist, collective_);
   	}
   	inline bool MPIOCollectiveMetadataRead::isCollective() const {
   	return collective_;
   	}
   	inline MPIOCollectiveMetadataRead::MPIOCollectiveMetadataRead(const FileAccessProps& plist) {
   	detail::h5p_get_all_coll_metadata_ops(plist.getId(), &collective_);
   	}
   	inline MPIOCollectiveMetadataRead::MPIOCollectiveMetadataRead(bool collective)
   	: collective_(collective) {}
   	inline void MPIOCollectiveMetadataWrite::apply(const hid_t plist) const {
   	detail::h5p_set_coll_metadata_write(plist, collective_);
   	}

I'm not sure which one is better.

There is an open PR to load Blosc: #980. We haven't made it past the issues of how to find Blosc and/or Blosc2. But it's a short, nice PR that shows Approach 2.

Since you say it's sparse data related:

• I know there's work on adding sparse data support to HDF5, but can't find the RFC for it (there is one, but it doesn't look like the one I'm thinking of).
• HighFive assumes the data is a dense multi-dimensional array. If read and write cause problems, you can try to first use read_raw and write_raw (they're essentially very thin wrappers around H5{A,D}{read,write}).

Please feel free to ask additional questions.

[AlexHenderson]

Hi @1uc ,

Thanks for this, I'll take a look.

My plan is to write a sparse filter, but that's really just a compression filter. Sparse data is on the HDF roadmap, but that isn't until version 2.0, so quite some way off. https://github.com/HDFGroup/hdf5/blob/develop/README.md.

I'd like to use HighFive because it makes using HDF5 in C++ very simple, much like h5py in Python. The HDF5 C++ interface is really a very thin wrapper around C, so has many of the same complexity and memory management issues. I wasn't going to write the filter in HighFive, only use HighFive as a platform to test it works (or doesn't).

I saw the PR #980 , but that looks to bake-in the blosc filter as a default. I would prefer to keep the default as the same default used by HDFGroup on the principle of 'least surprise'. However, the ability to drop in dynamic filters is very powerful, and would actually solve that person's problem. Simply drop in blosc at runtime.

What I'm not sure about is how to call an external filter. There are many to choose from (https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md), but how to set a filter and its properties is a little unclear. I wasn't sure if HighFive incorporated the H5Pset_filter interface.

Thanks, Alex