Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the option for using blosc filter #980

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add the option for using blosc filter #980

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,11 @@ target_link_libraries(HighFive INTERFACE HighFive::Include)
if(HIGHFIVE_FIND_HDF5)
find_package(HDF5 REQUIRED)
target_link_libraries(HighFive INTERFACE HDF5::HDF5)

# I install hdf5-blosc to the default path
set(blosc_filter_DIR /usr/local/lib/)
target_link_libraries(HighFive INTERFACE blosc_filter)
target_link_directories(HighFive INTERFACE /usr/local/include/)
Comment on lines +105 to +108
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will have to fix that.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since blosc is an extra plugin of hdf5, it is necessary to compile and link this external libraries. hdf5-blosc can not be found using find_package macro since it is packaged too simple here. I haven't figured out an elegant way to link this library.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is that hard coded path to your system is not the valid way to give access to blosc.

endif()

if(HDF5_IS_PARALLEL)
Expand Down
11 changes: 11 additions & 0 deletions include/highfive/H5PropertyList.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -556,6 +556,17 @@ class Shuffle {
void apply(hid_t hid) const;
};

/// \implements PropertyInterface
class Blosc {
public:
explicit Blosc(unsigned level, unsigned shuffle, unsigned compressor);

private:
friend DataSetCreateProps;
void apply(hid_t hid) const;
unsigned _cd_values[7];
};

/// \brief When are datasets allocated?
///
/// The precise time of when HDF5 requests space to store the dataset
Expand Down
10 changes: 10 additions & 0 deletions include/highfive/bits/H5PropertyList_misc.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,16 @@ inline void Shuffle::apply(const hid_t hid) const {
detail::h5p_set_shuffle(hid);
}

inline Blosc::Blosc(unsigned int level, unsigned int shuffle, unsigned int compressor) {
_cd_values[4] = level;
_cd_values[5] = shuffle;
_cd_values[6] = compressor;
}

inline void Blosc::apply(const hid_t hid) const {
detail::h5p_set_blosc(hid, _cd_values);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function likely can't be called twice. Therefore, if two property lists with Blosc are created I suspect a misleading error message will be printed. It's unclear if it'll work, because register_blosc seems to unconditionally return 1; (despite defining and setting a retval).

}

inline AllocationTime::AllocationTime(H5D_alloc_time_t alloc_time)
: _alloc_time(alloc_time) {}

Expand Down
15 changes: 15 additions & 0 deletions include/highfive/bits/h5p_wrapper.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,21 @@ inline herr_t h5p_set_szip(hid_t plist_id, unsigned options_mask, unsigned pixel
return err;
}

#include "blosc_filter.h"

inline herr_t h5p_set_blosc(hid_t plist_id, const unsigned int *cd_values) {
char *version, *date;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are both leaked, because register_blosc uses strdup which requires us to call free.

if (register_blosc(&version, &date) < 0) {
HDF5ErrMapper::ToException<PropertyException>("Blosc filter unavailable.");
}

herr_t err = H5Pset_filter(plist_id, FILTER_BLOSC, H5Z_FLAG_OPTIONAL, 7, cd_values);
if (err < 0) {
HDF5ErrMapper::ToException<PropertyException>("Error setting blosc property");
}
return err;
}
Comment on lines +244 to +255
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@1uc I don't think this is how you think how your wrapper, right?
We will need to put this helper in an other place?

Copy link
Author

@LauZanMo LauZanMo Apr 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unlike szip, a built-in filter in hdf5, blosc filter must be registered before use. Perhaps a register wrapper should be added here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I'd prefer splitting it differently:

inline herr_t h5p_set_filter(....) {
    herr_t err = H5Pset_filter(...);
    if (err < 0) {
        HDF5ErrMapper::ToException<PropertyException>("Error setting setting filter.");
    }
    return err;
}

Then in Blosc we do the other half:

inline void Blosc::apply(const hid_t hid) const {
    char *version, *date;
    if (register_blosc(&version, &date) < 0) {
        HDF5ErrMapper::ToException<PropertyException>("Blosc filter unavailable.");
    }
    detail::h5p_set_filter(plist_id, FILTER_BLOSC, H5Z_FLAG_OPTIONAL, 7, cd_values);
}

If you want the error message to contain the word "blosc" when set_filter fail, one would need to catch and rethrow with a different error message.


inline herr_t h5p_set_shuffle(hid_t plist_id) {
herr_t err = H5Pset_shuffle(plist_id);
if (err < 0) {
Expand Down
6 changes: 5 additions & 1 deletion src/examples/create_dataset_double.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,12 @@ int main(void) {
// Define the size of our dataset: 2x6
std::vector<size_t> dims{2, 6};

HighFive::DataSetCreateProps props;
props.add(HighFive::Chunking({2,2}));
props.add(HighFive::Blosc(4, 1, BLOSC_BLOSCLZ));

// Create the dataset
DataSet dataset = file.createDataSet<double>("dset", DataSpace(dims));
DataSet dataset = file.createDataSet<double>("dset", DataSpace(dims), props);

double data[2][6] = {{1.1, 2.2, 3.3, 4.4, 5.5, 6.6},
{11.11, 12.12, 13.13, 14.14, 15.15, 16.16}};
Expand Down