Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add compressor option to write image #161

Merged
merged 10 commits into from
Feb 8, 2022
Merged

add compressor option to write image #161

merged 10 commits into from
Feb 8, 2022

Conversation

satra
Copy link
Contributor

@satra satra commented Jan 24, 2022

this proposes a change to the write_image api to allow compressors. this is now more general to pass any options to the create dataset function and should handle multilevel options.

closes #160

  • Write some tests

satra and others added 2 commits January 24, 2022 13:28
this proposes a change to the write_image api to allow compressors
@codecov
Copy link

codecov bot commented Jan 24, 2022

Codecov Report

Merging #161 (0d9e868) into master (3e3327c) will increase coverage by 0.46%.
The diff coverage is 90.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #161      +/-   ##
==========================================
+ Coverage   83.53%   83.99%   +0.46%     
==========================================
  Files          12       12              
  Lines        1354     1362       +8     
==========================================
+ Hits         1131     1144      +13     
+ Misses        223      218       -5     
Impacted Files Coverage Δ
ome_zarr/writer.py 98.17% <90.00%> (+1.85%) ⬆️
ome_zarr/format.py 98.17% <0.00%> (+1.35%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3e3327c...0d9e868. Read the comment docs.

@joshmoore
Copy link
Member

Thanks for this, @satra. Could you sign and return the OME CLA as described in https://ome-contributing.readthedocs.io/en/latest/cla.html?

Copy link
Member

@joshmoore joshmoore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation seems simple enough, though we should probably get it integrated into a test.

As a side note, I do remember, however, not being completely happy with the chunks argument since conceivably one might want different settings per pyramid level. The same might apply here.

I don't know if you, @satra, or anyone else at this point has a clear alternative that would keep this method cleaner. Options that come to mind include:

  • passing all arguments as key words similar to fsspec's storage_options
  • and/or passing a callback function which review is given the path, the dataset, as well as the resolution level

Happy to keep pondering those after this PR though.

@sbesson
Copy link
Member

sbesson commented Jan 26, 2022

Brainstorming on the API questions raised in #161 (review), a solution along the lines of storage_options feels appealing to me as this would allow any of the options available in create_dataset to be specified via this library.

Do we expect the dataset level to be the finer level of granularity for these storage options? In this case, I assume an option would be either to support a simple dictionary (and use it for each dataset) or a list of dictionaries of size matching the size of pyramid which would allowing to specify options at the individual dataset level.

@satra
Copy link
Contributor Author

satra commented Feb 4, 2022

@sbesson - updated this to use storage_options.

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @satra. We are making progress towards ome-zarr 0.3.0 which should include OME-NGFF 0.4 support. It would be great to make this new storage options available at the same time.

At minimum, could a unit test be added that covers the compression scenario? As a bonus, it looks like your storage_options also cover the per-resolution chunks feature mentioned in #161 (review). I suspect we will want some tests covering this scenario but this can happen as a follow-up PR depending on time.
Also could you return the CLA as mentioned in #161 (comment).

else storage_options[path]
)
if "chunks" not in options:
options["chunks"] = chunks
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This raises the question of whether we would like to deprecate the explicit chunks argument and only rely on the storage_options dictionary moving forward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it's deprecated, i would suggest giving it a couple of cycles and perhaps add a deprecation warning with the next release. but leave this up to you.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, we can gain knowledge about the storage_options in the upcoming 0.3.x and schedule the deprecation of chunks for 0.4. I'll create an issue

Copy link
Member

@sbesson sbesson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the test @satra. All looks good from my side. Merging and we will aim to get this as a public release in the upcoming days

else storage_options[path]
)
if "chunks" not in options:
options["chunks"] = chunks
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough, we can gain knowledge about the storage_options in the upcoming 0.3.x and schedule the deprecation of chunks for 0.4. I'll create an issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add compressor to write_image
3 participants