Skip to content
This repository has been archived by the owner on Jun 21, 2022. It is now read-only.

Add zstd compression read support for uproot #421

Merged
merged 8 commits into from
Dec 16, 2019

Conversation

oshadura
Copy link
Contributor

@oshadura oshadura commented Dec 16, 2019

Locally I have green tests:
Ran 106 tests in 33.114s
OK

appveyor.yml Outdated Show resolved Hide resolved
Copy link
Member

@jpivarski jpivarski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, this looks perfect! Highly approved! (Oh, two mistakes; but after that, highly approved.)

  • Python zstd library is an optional dependency.
  • Oh, I found an error: the installation instructions for conda should be changed. (See compressed.py:89 inline comment.)
  • It's in the tests-require and manually loaded in Travis (like the others).
  • Test samples with the new compression algorithm (HZZ and Zmumu).
  • Unit tests use the new test samples, comparing detailed TTree values (test_compression.py, content1 and content2.
  • sample4version.C is ready to make new ROOT files with zstd, but that will wait until ROOT 6.20 comes out.
  • Help strings have been updated.
  • New constants in const.py.

The failure of the AppVeyor tests is due to appveyor.yml:53 (see inline comment).

uproot/source/compressed.py Outdated Show resolved Hide resolved
appveyor.yml Outdated Show resolved Hide resolved
@oshadura
Copy link
Contributor Author

@jpivarski I will add zstd for writing as soon we will have release.

@jpivarski
Copy link
Member

@oshadura Writing with zstd should probably be a new pull request, and it may be after the 3.11.0 release.

When you do add writing with zstd, be sure to ping @reikdas so he can verify that nothing was missed.

@jpivarski
Copy link
Member

@oshadura I pushed a correction to your branch, but it hasn't shown up here yet (for me): https://github.com/oshadura/uproot/commits/add-zstd-compression-support

When it does (and tests pass), I'll merge this unless you tell me not to.

@jpivarski
Copy link
Member

@oshadura Could you try pulling in your branch, verifying that my changes to uproot/source/compressed.py are there (where I dropped the -c XYZ option in the install messages), and maybe make some trivial change to push it here? I don't know why it isn't showing up.

@chrisburr
Copy link
Member

I was looking at adding the conda-forge package and I noticed there are two different sets of Python bindings. Both are linked from facebook's webpage:

Is there a reason for preferring zstd?

@oshadura
Copy link
Contributor Author

@jpivarski

I was looking at adding the conda-forge package and I noticed there are two different sets of Python bindings. Both are linked from facebook's webpage:

* https://pypi.org/project/zstd/: Currently used

* https://pypi.org/project/zstandard/: More downloads/GitHub activity and already on conda-forge

Is there a reason for preferring zstd?

Thanks, let me check!

@oshadura
Copy link
Contributor Author

https://pypi.org/project/zstd/

@jpivarski @chrisburr thanks for checking, zstandard package is definitely better! (it supports much more functionality, such as dictionary generation and streaming API). I updated PR accordingly.

@jpivarski
Copy link
Member

Okay, great! When the tests pass, should I merge this, @oshadura?

@oshadura
Copy link
Contributor Author

Okay, great! When the tests pass, should I merge this, @oshadura?

Yes, please!

@oshadura oshadura force-pushed the add-zstd-compression-support branch from 1d8af06 to 83169d9 Compare December 16, 2019 18:17
@jpivarski
Copy link
Member

After this last force-push?

oshadura force-pushed the oshadura:add-zstd-compression-support branch from 1d8af06 to 83169d9 2 minutes ago

@oshadura
Copy link
Contributor Author

@jpivarski yeap, now I am ready I think.

@jpivarski jpivarski merged commit 6e89ce6 into scikit-hep:master Dec 16, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants