Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updated embedded libraries: SZ3 #289

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

Updated embedded libraries: SZ3 #289

wants to merge 15 commits into from

Conversation

t20100
Copy link
Member

@t20100 t20100 commented Jan 19, 2024

This PR aims at upgrading SZ3 to the latest release: https://github.com/szcompressor/SZ3/releases/tag/v3.1.8

@t20100 t20100 requested a review from vasole January 19, 2024 08:44
@t20100
Copy link
Member Author

t20100 commented Jan 19, 2024

The failing test is the one reading an already exisiting file, so that was compressed with a previous version of the filter.
The round-trip test pass.
So the new version of the SZ3 filter cannot read file generated with previous versions of the filter....

@vasole
Copy link
Member

vasole commented Jan 19, 2024

So the new version of the SZ3 filter cannot read file generated with previous versions of the filter....

I guess we'll have to check if the SZ3 developers are aware of it.

@vasole vasole mentioned this pull request Jan 19, 2024
Open
@t20100 t20100 marked this pull request as draft February 2, 2024 10:47
@t20100 t20100 changed the title Updated embedded libraries: SZ3 v3.1.8 Updated embedded libraries: SZ3 Feb 9, 2024
@t20100
Copy link
Member Author

t20100 commented Feb 9, 2024

This Pr is pending for a new release of SZ3 (see szcompressor/SZ3#50 (comment))

@t20100 t20100 added this to the Next release milestone Feb 9, 2024
@t20100
Update embedded PyTables code to v3.10.1
9feaf95
@t20100
Remove src/c-blosc
3be12b8
@t20100
Squashed 'src/c-blosc/' content from commit 43377c44
2395a02 
git-subtree-dir: src/c-blosc
git-subtree-split: 43377c44427868b1d2d9b11b43d6ade2ba75216c
@t20100
Merge commit '2395a02497a7f25ea94b904d015034db9654d63d' as 'src/c-blosc'
0785e07
@t20100
Squashed 'src/c-blosc2/' changes from 7424ecfb6..394c862c8
8e226b0 
841c6ae72 Getting ready for release 2.15.1
843bb60d9 Undo the warning fix
253c2cc03 Fix a warning on Linux
bb66cce09 Internal LZ4 sources bumped to 1.10.0.
7be3f6fbf load_lib: allow direct loading of library by name
3669c5c4c Add b2nd_nans method #624
0ec41df74 Add info about filename extensions
f3aea9edc Add a check of architecture to select simd probing implementation
5ff43ff87 Add stubs and runtime checks for ALTIVEC shuffle/bitshuffle implementation
64e1ce8a4 Add stubs and runtime checks for NEON shuffle/bitshuffle implementation
bf437c382 Add stubs and runtime checks for AVX512 bitshuffle implementation
9f99b3af6 Add stubs and runtime checks for SSE2 shuffle/bitshuffle implementation
a675b071f Add stubs and runtime checks for AVX2 shuffle/bitshuffle implementation
2a07f3457 Do not pass `-m` flags when compiling `shuffle.c`
dc5d651c3 Post 2.15.0 release actions done

git-subtree-dir: src/c-blosc2
git-subtree-split: 394c862c893354241c4e9b95a3142c5af7c90b9d
@t20100
Merge commit '8e226b0f250c70c44a0cb5a958bb7bd5e115e6b3' into updates
ba8952e
@t20100
remove src/fcidecomp
4239c47
@t20100
Add fcidecomp v2.1.1 from tarball
8b5b4f7
@t20100
update embedded libraries and filters doc
3305a76
@t20100
update c-blosc2 macros
9db17b9
@t20100
Squashed 'src/SZ3/' changes from 4bbe9df7e..b3dab4018
023828e 
b3dab4018 Merge pull request silx-kit#60 from szcompressor/newapi
d70c22264 update readme
35115beeb update readme
352c1a0f6 update compress api
fe6eef7e3 update compress api
75c906dda update compress api
ad47f20d2 update compress api
52b4f439c update compress api
7a5447c79 update compress api
cb87901dd Merge remote-tracking branch 'origin/newapi' into newapi
c1d6504e1 update hdf5 filter
533292f9f Merge branch 'refs/heads/master' into newapi
b3dac0c65 update readme
3f478178d Merge pull request silx-kit#56 from juntyr/32bit-hash
f8fc2cb33 fix int overflow in quantization
e52a2c30f Add support for 32bit systems by falling back to std::unordered_map
f6cae37f5 add link to fortran api
90c66bed1 modify MDZ CR calculation to include the first frame which is stored losslessly.
09494b13c fix bug in MDZ for 1D inputs
dde630245 add noprediction compressor
47593621f bugfix
fdd6dee6e API rewrite data version check
a887550ed API rewrite data version check
5bab19c5d API rewrite data version check
1ea5d5b07 API rewrite data version check
e78f4832f Add annotation
be68d6cbe bug fix for openmp
800c3248e bug fix for interp
d06710a55 bug fix for interp
7fc2cb519 bug fix for MDZ
3568914cd Merge remote-tracking branch 'origin/master'
5f835fb74 bug fix for MDZ
be68d645b Merge pull request silx-kit#47 from robertu94/master
e308ebf85 Merge branch 'master' of github.com:szcompressor/sz3
6198db715 default initalize config
162674f87 Merge pull request silx-kit#46 from mrmbernardi/patch-1
f834756f9 Fix SZFastFrontend::size_est() returning undefined value
13e79072e config bugfix
0ac567a76 Merge pull request silx-kit#45 from szcompressor/hdf5
d6beca327 add config support to hdf5 filter
05965ac2b update 1d lorenzo fast mode
d2a03eae4 add lorenzo 1D fast mode
fa9a7f179 Merge pull request silx-kit#44 from szcompressor/namespace
2437574dc change namespace from SZ to SZ3, to avoid conflict with SZ2
3236a6eb2 Merge branch 'huffman'
5995fa9f2 update HDF5 filter
1fc15171c update HDF5 filter
aadaaeb90 change namespace from SZ to SZ3, to avoid conflict with SZ2
e1af6bea2 change namespace from SZ to SZ3, to avoid conflict with SZ2
bdc4b503b new huffman
9aee288dd Merge pull request silx-kit#43 from mrmbernardi/decompress_output
ab7896ec4 update stateNum in preprocess_encode
44839697c Decompress function returns output matching original_dtype
f420554d3 update readme
31d7cbb6a add annotation for interfaces
d0606d3ae Merge pull request silx-kit#41 from szcompressor/mdz
3443ae347 update readme
1219a336d disable SZ3_DEBUG_TIMINGS by default
09d43dee5 clear runtime output for MDZ
f051a8361 support 3D input for MDZ
89f3b7c78 bugfix in smoke test
5a1cf1aad Merge remote-tracking branch 'origin/master'
40e9b3de5 bugfix in smoke test
140e09e9c remove the print information (OpenMP threads = ), which might be annoying and seems unneeded.
1e1dc102e Merge pull request silx-kit#38 from aumuell/uint8
9b2bf8ad5 fix missing uint8_t with GCC 13
f7d865e6e Merge pull request silx-kit#37 from robertu94/hdf5_fix
73674f3db link to MPI when hdf5 is parallel
13f835d54 openmp bugfix
318fe953f openmp bugfix
612b4993d update pysz
bce7f1202 Merge pull request silx-kit#32 from exmakhina/master
62d01de0b python: load library from a sensible default path, if not provided
ce7fb031e change sprintf to snprintf
1b1abcf39 interp+lorenzo sampling bugfix
9f396935d bugfix

git-subtree-dir: src/SZ3
git-subtree-split: b3dab4018425803a55d8073dc55dade7fa46b7b4
@t20100
Merge commit '023828e42c4630463a777d54c03257006800ff95' into update-sz3
45cb691
@t20100
update header to match sz3 v3.2.0
2ba2424
@t20100
simplify include dirs
c51e24a
@t20100
update documentation of embedded SZ3 version
06bd935
@t20100
Copy link
Member Author

t20100 commented Aug 28, 2024

Updated to use SZ3 v3.2.0 which checks that the data was compressed with a compatible version of the library.

Unfortunately, the cd_values format has changed from a couple of arguments to the dump of an instance of a Config C++ class (conversion code).

This makes reading a HDF5 dataset compressed with a previous version of SZ3 abort the process because it cannot read the Config object. This defeats the check of the version by the library.
For instance, test/test.py fails with:

testSZ3 (__main__.TestHDF5PluginRead)
  Test reading and witing SZ3 compressed data ... terminate called after throwing an instance of 'std::length_error'
    what():  vector::_M_default_append
  start H5Z_filter_sz3

Also, when writing a new HDF5 dataset, the current passed arguments must be changed, so as it is the hdf5plugin.test.TestHDF5PluginRW.testSZ3 test fails with:

start H5Z_sz3_set_local
terminate called after throwing an instance of 'std::length_error'
  what():  vector::_M_default_append
Aborted

It works when setting SZ3.filter_options = ().
To make it work means re-implementing and the dump of the Config object to integers here in Python.
Since this mechanism was not developed to be use outside of C++, I'm -1 to duplicate it here.

Finally, it would be good to check that the Config read/write works when writing with one endianess and reading with another.

@t20100 t20100 removed this from the 5.0.0 milestone Aug 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vasole vasole Awaiting requested review from vasole

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@t20100 @vasole