Finalize the compression support. #399
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This relies on the HDF5 capability to dynamically load compression filters. Note that a compression filter is just a subcase of filters. The primary user-visible changes are as follows: 1. Add a standard header "netcdf_filter.h" that defines the necessary API extensions 2. Modify ncgen to support two new special attributes "_Filter_ID" and "_Filter_Parameters" so that compression can be turned on when creating a file using ncgen. 4. Add a detailed description of filtering support to the user's guide; see the file filters.md 5. Add a test case directory for this: nc_test4/filter_test. It is fragile and a ./configure flags (-enable-filter-test) is defined (default disabled) to shut this off this test to avoid spurious 'make check' failures. Note that the HDF5 documentation is not up-to-date, so much of what is encoded here comes from examining the actual code in the file H5PL.c in the HDF5 source code.
1. Allow nccopy to apply filters, especially on the output file. This provides a third way to do this other than using ncgen or programatically 2. Make sure that even if the filter code is not available, it is possible to see the filter id and parameters for variables using e.g ncdump -hs. 3. Fix bug in nccopy so that the input file does not necessarily have to be netcdf-4. 4. At last minute decided to change to using a single "_Filter" attribute for ncgen 5. Added a test to tst_filter.sh to generate C code using ncgen.
|
Add new capabilities to filter code:
|
a linux variant system, I made some changes to (one hopes) incorporate some improvements in documenting and building this test.
to docs/filter.md 2. Moved location of filter.md in documentation 3. Add a template file as the basis for building new filters. 4. Did some test case cleanup
2. Factored out the parameter string parsing for ncgen and nccopy int libdispatch/dfilter.c + include/ncfilter.h 3. Allow a parameter string to use constant types other than unsigned int. See docs/filters.md for details. 4. Moved the old content of include/netcdf_filter.h into include/netcdf.h and removed include/netcdf_filter.h as no longer needed. 5. Force the test filter (bzip2) in nc_test4/filter_test to be built using BUILT_SOURCES.
|
Working on this next; it has been open for a long time and needs merged. Will work on the conflicts. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pr finalizes the netcdf-c compression support.
It relies on the HDF5 capability to
dynamically load compression filters.
Note that a compression filter is just
a special case of a filter.
The primary user-visible changes are as follows:
the necessary API extensions
"_Filter_ID" and "_Filter_Parameters" so that compression
can be turned on when creating a file using ncgen.
to the user's guide; see the file filters.md
It is fragile and a ./configure flags (-enable-filter-test)
is defined (default disabled) to shut this off this test
to avoid spurious 'make check' failures.
Note that the HDF5 documentation is not up-to-date, so
much of what is encoded here comes from examining the
actual code in the file H5PL.c in the HDF5 source code.