Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation updates. #93

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions NAMESPACE
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,10 @@ export(zarr_open_array)
export(zarr_open_group)
export(zarr_save_array)
export(zb_slice)
importFrom(R6,R6Class)
importFrom(memoise,memoise)
importFrom(memoise,timeout)
importFrom(qs,lz4_compress_raw)
importFrom(qs,lz4_decompress_raw)
importFrom(qs,zstd_compress_raw)
importFrom(qs,zstd_decompress_raw)
keller-mark marked this conversation as resolved.
Show resolved Hide resolved
1 change: 1 addition & 0 deletions R/array-nested.R
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ zero_based_to_one_based <- function(selection, shape) {
#' Represents a multi-dimensional array that can be
#' accessed and subsetted via list of Slice instances.
#' @rdname NestedArray
#' @importFrom R6 R6Class
#' @export
NestedArray <- R6::R6Class("NestedArray",
private = list(
Expand Down
156 changes: 17 additions & 139 deletions R/creation.R
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ require_parent_group <- function(
#' Primary compressor.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a ton of clean up in here. Nothing should have changed but the duplication is basically all gone. A couple check issues are taken care of here too.

#' @param fill_value : object
#' Default value to use for uninitialized portions of the array.
#' @param order : {'C', 'F'}, optional
#' @param order : \code{'C', 'F'}, optional
#' Memory layout to be used within each chunk.
#' @param overwrite : bool, optional
#' If True, erase all data in `store` prior to initialisation.
Expand All @@ -256,7 +256,7 @@ require_parent_group <- function(
#' Sequence of filters to use to encode chunk data prior to compression.
#' @param object_codec : Codec, optional
#' A codec to encode object arrays, only needed if dtype=object.
#' @param dimension_separator : {'.', '/'}, optional
#' @param dimension_separator : \code{'.', '/'}, optional
#' Separator placed between the dimensions of a chunk.
#' @keywords internal
init_array <- function(
Expand Down Expand Up @@ -307,15 +307,7 @@ init_array <- function(

#' Initialize a group store. Note that this is a low-level function and there should be no
#' need to call this directly from user code.
#' @param store : Store
#' A mapping that supports string keys and byte sequence values.
#' @param overwrite : bool, optional
#' If True, erase all data in `store` prior to initialisation.
#' @param path : string, optional
#' Path under which array is stored.
#' @param chunk_store : Store, optional
#' Separate storage for chunks. If not provided, `store` will be used
#' for storage of both chunks and metadata.
#' @inheritParams init_array
#' @keywords internal
init_group <- function(
store,
Expand All @@ -339,35 +331,14 @@ init_group <- function(


#' Create an empty array
#' @param shape : int or tuple of ints
#' Array shape.
#' @inheritParams init_array
#' @param chunks : int or tuple of ints, optional
#' Chunk shape. If True, will be guessed from `shape` and `dtype`. If
#' False, will be set to `shape`, i.e., single chunk for the whole array.
#' If an int, the chunk size in each dimension will be given by the value
#' of `chunks`. Default is True.
#' @param dtype : string or dtype, optional
#' NumPy dtype.
#' @param compressor : Codec, optional
#' Primary compressor.
#' @param fill_value : object
#' Default value to use for uninitialized portions of the array.
#' @param order : {'C', 'F'}, optional
#' Memory layout to be used within each chunk.
#' @param store : MutableMapping or string
#' Store or path to directory in file system or name of zip file.
#' @param synchronizer : object, optional
#' Array synchronizer.
#' @param overwrite : bool, optional
#' If True, delete all pre-existing data in `store` at `path` before
#' creating the array.
#' @param path : string, optional
#' Path under which array is stored.
#' @param chunk_store : MutableMapping, optional
#' Separate storage for chunks. If not provided, `store` will be used
#' for storage of both chunks and metadata.
#' @param filters : sequence of Codecs, optional
# Sequence of filters to use to encode chunk data prior to compression.
#' @param cache_metadata : bool, optional
#' If True, array configuration metadata will be cached for the
#' lifetime of the object. If False, array metadata will be reloaded
Expand All @@ -379,10 +350,6 @@ init_group <- function(
#' to all attribute read operations.
#' @param read_only : bool, optional
#' True if array should be protected against modification.
#' @param object_codec : Codec, optional
#' A codec to encode object arrays, only needed if dtype=object.
#' @param dimension_separator : {'.', '/'}, optional
#' Separator placed between the dimensions of a chunk.
#' @param write_empty_chunks : bool, optional
#' If True (default), all chunks will be stored regardless of their
#' contents. If False, each chunk is compared to the array's fill value
Expand Down Expand Up @@ -449,7 +416,7 @@ zarr_create <- function(
}

#' Create an array filled with NAs.
#' @param shape : int or tuple of ints
#' @inheritParams zarr_create
#' @param ... The params of zarr_create()
#' @returns ZarrArray
#' @export
Expand All @@ -470,7 +437,7 @@ zarr_create_array <- function(data, ...) {
}

#' Create an array filled with zeros.
#' @param shape : int or tuple of ints
#' @inheritParams zarr_create
#' @param ... The params of zarr_create()
#' @returns ZarrArray
#' @export
Expand All @@ -479,22 +446,8 @@ zarr_create_zeros <- function(shape, ...) {
}

#' Create a group.
#' @param store : MutableMapping or string, optional
#' Store or path to directory in file system.
#' @param overwrite : bool, optional
#' If True, delete any pre-existing data in `store` at `path` before
#' creating the group.
#' @param chunk_store : MutableMapping, optional
#' Separate storage for chunks. If not provided, `store` will be used
#' for storage of both chunks and metadata.
#' @param cache_attrs : bool, optional
#' If True (default), user attributes will be cached for attribute read
#' operations. If False, user attributes are reloaded from the store prior
#' to all attribute read operations.
#' @param synchronizer : object, optional
#' Array synchronizer.
#' @param path : string, optional
#' Group path within store.
#' @inheritParams init_array
#' @inheritParams zarr_create
#' @returns ZarrGroup
#' @export
zarr_create_group <- function(
Expand Down Expand Up @@ -529,23 +482,8 @@ zarr_create_group <- function(
}

#' Open a group using file-mode-like semantics.
#' @param store : MutableMapping or string, optional
#' Store or path to directory in file system or name of zip file.
#' @param mode : {'r', 'r+', 'a', 'w', 'w-'}, optional
#' Persistence mode: 'r' means read only (must exist); 'r+' means
#' read/write (must exist); 'a' means read/write (create if doesn't
#' exist); 'w' means create (overwrite if exists); 'w-' means create
#' (fail if exists).
#' @param cache_attrs : bool, optional
#' If True (default), user attributes will be cached for attribute read
#' operations. If False, user attributes are reloaded from the store prior
#' to all attribute read operations.
#' @param synchronizer : object, optional
#' Array synchronizer.
#' @param path : string, optional
#' Group path within store.
#' @param chunk_store : MutableMapping or string, optional
#' Store or path to directory in file system or name of zip file.
#' @inheritParams zarr_open
#' @inheritParams zarr_create
#' @param storage_options : dict
#' If using an fsspec URL to create the store, these will be passed to
#' the backend implementation. Ignored otherwise.
Expand Down Expand Up @@ -617,66 +555,10 @@ zarr_open_group <- function(
}

#' Open an array using file-mode-like semantics.
#' @param store : MutableMapping or string, optional
#' Store or path to directory in file system or name of zip file.
#' @param storage_options : dict
#' If using an fsspec URL to create the store, these will be passed to
#' the backend implementation. Ignored otherwise.
#' @param mode : {'r', 'r+', 'a', 'w', 'w-'}, optional
#' Persistence mode: 'r' means read only (must exist); 'r+' means
#' read/write (must exist); 'a' means read/write (create if doesn't
#' exist); 'w' means create (overwrite if exists); 'w-' means create
#' (fail if exists).
#' @param shape : int or tuple of ints
#' Array shape.
#' @param chunks : int or tuple of ints, optional
#' Chunk shape. If True, will be guessed from `shape` and `dtype`. If
#' False, will be set to `shape`, i.e., single chunk for the whole array.
#' If an int, the chunk size in each dimension will be given by the value
#' of `chunks`. Default is True.
#' @param dtype : string or dtype, optional
#' NumPy dtype.
#' @param compressor : Codec, optional
#' Primary compressor.
#' @param fill_value : object
#' Default value to use for uninitialized portions of the array.
#' @param order : {'C', 'F'}, optional
#' Memory layout to be used within each chunk.
#' @param store : MutableMapping or string
#' Store or path to directory in file system or name of zip file.
#' @param synchronizer : object, optional
#' Array synchronizer.
#' @param overwrite : bool, optional
#' If True, delete all pre-existing data in `store` at `path` before
#' creating the array.
#' @param path : string, optional
#' Path under which array is stored.
#' @param chunk_store : MutableMapping, optional
#' Separate storage for chunks. If not provided, `store` will be used
#' for storage of both chunks and metadata.
#' @param filters : sequence of Codecs, optional
# Sequence of filters to use to encode chunk data prior to compression.
#' @param cache_metadata : bool, optional
#' If True, array configuration metadata will be cached for the
#' lifetime of the object. If False, array metadata will be reloaded
#' prior to all data access and modification operations (may incur
#' overhead depending on storage and data access pattern).
#' @param cache_attrs : bool, optional
#' If True (default), user attributes will be cached for attribute read
#' operations. If False, user attributes are reloaded from the store prior
#' to all attribute read operations.
#' @param object_codec : Codec, optional
#' A codec to encode object arrays, only needed if dtype=object.
#' @param dimension_separator : {'.', '/'}, optional
#' Separator placed between the dimensions of a chunk.
#' @param write_empty_chunks : bool, optional
#' If True (default), all chunks will be stored regardless of their
#' contents. If False, each chunk is compared to the array's fill value
#' prior to storing. If a chunk is uniformly equal to the fill value, then
#' that chunk is not be stored, and the store entry for that chunk's key
#' is deleted. This setting enables sparser storage, as only chunks with
#' non-fill-value data are stored, at the expense of overhead associated
#' with checking the data of each chunk.
#' @inheritParams init_array
#' @inheritParams zarr_create
#' @inheritParams zarr_open
#' @inheritParams zarr_open_group
#' @returns ZarrArray
#' @export
zarr_open_array <- function(
Expand Down Expand Up @@ -784,8 +666,7 @@ zarr_open_array <- function(
}

#' Convenience function to save a ZarrArray to the local file system.
#' @param store : MutableMapping or string
#' Store or path to directory in file system or name of zip file.
#' @inheritParams init_array
#' @param arr : ZarrArray
#' The array with data to save.
#' @param ... Additional arguments to pass to zarr_create_array().
Expand All @@ -797,15 +678,12 @@ zarr_save_array <- function(store, arr, ...) {
}

#' Convenience function to open a group or array using file-mode-like semantics.
#' @param store : MutableMapping or string, optional
#' Store or path to directory in file system or name of zip file.
#' @param mode : {'r', 'r+', 'a', 'w', 'w-'}, optional
#' @inheritParams init_array
#' @param mode : \code{'r', 'r+', 'a', 'w', 'w-'}, optional
#' Persistence mode: 'r' means read only (must exist); 'r+' means
#' read/write (must exist); 'a' means read/write (create if doesn't
#' exist); 'w' means create (overwrite if exists); 'w-' means create
#' (fail if exists).
#' @param path : str or NA, optional
#' The path within the store to open.
#' @param ... Additional arguments to pass to zarr_open_array or zarr_open_group.
#' @returns ZarrArray or ZarrGroup
#' @export
Expand Down
10 changes: 6 additions & 4 deletions R/numcodecs.R
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Codec <- R6::R6Class("Codec",
#' Class representing a ZSTD compressor

#' @rdname ZstdCodec
#' @importFrom qs zstd_compress_raw zstd_decompress_raw
#' @export
ZstdCodec <- R6::R6Class("ZstdCodec",
inherit = Codec,
Expand All @@ -61,7 +62,7 @@ ZstdCodec <- R6::R6Class("ZstdCodec",
#' @return Compressed data.
encode = function(buf, zarr_arr) {
# Reference: https://github.com/traversc/qs/blob/84e30f4/R/RcppExports.R#L16
result <- qs::zstd_compress_raw(buf, self$level)
result <- zstd_compress_raw(buf, self$level)
return(result)
},
#' @description
Expand All @@ -70,7 +71,7 @@ ZstdCodec <- R6::R6Class("ZstdCodec",
#' @param zarr_arr The ZarrArray instance.
#' @return Un-compressed data.
decode = function(buf, zarr_arr) {
result <- qs::zstd_decompress_raw(buf)
result <- zstd_decompress_raw(buf)
return(result)
},
#' @description
Expand All @@ -92,6 +93,7 @@ ZstdCodec <- R6::R6Class("ZstdCodec",
#' Class representing a LZ4 compressor
#'
#' @rdname Lz4Codec
#' @importFrom qs lz4_compress_raw lz4_decompress_raw
#' @export
Lz4Codec <- R6::R6Class("Lz4Codec",
inherit = Codec,
Expand All @@ -113,7 +115,7 @@ Lz4Codec <- R6::R6Class("Lz4Codec",
#' @return Compressed data.
encode = function(buf, zarr_arr) {
# Reference: https://github.com/traversc/qs/blob/84e30f4/R/RcppExports.R#L24
body <- qs::lz4_compress_raw(buf, self$acceleration)
body <- lz4_compress_raw(buf, self$acceleration)

# The compressed output includes a 4-byte header storing the original size
# of the decompressed data as a little-endian 32-bit integer.
Expand All @@ -133,7 +135,7 @@ Lz4Codec <- R6::R6Class("Lz4Codec",
decode = function(buf, zarr_arr) {
body <- buf[5:length(buf)]

result <- qs::lz4_decompress_raw(body)
result <- lz4_decompress_raw(body)
return(result)
},
#' @description
Expand Down
9 changes: 4 additions & 5 deletions R/slicing.R
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ Slice <- R6::R6Class("Slice",
)
)

#' Convenience function for the internal Slice class constructor.
#' Convenience function for the internal Slice R6 class constructor.
#' @param start The start index.
#' @param stop The stop index.
#' @param step The step size.
Expand All @@ -119,11 +119,10 @@ slice <- function(start, stop = NA, step = NA, zero_based = FALSE) {
))
}

#' Convenience function for the internal Slice class constructor
#' Convenience function for the internal Slice R6 class constructor
#' with zero-based indexing and exclusive stop index.
#' @param start The start index.
#' @param stop The stop index.
#' @param step The step size.
#' @inheritParams slice
#' @inherit slice return
#' @export
zb_slice <- function(start, stop = NA, step = NA) {
return(slice(start, stop, step, zero_based = TRUE))
Expand Down
5 changes: 3 additions & 2 deletions R/stores.R
Original file line number Diff line number Diff line change
Expand Up @@ -351,6 +351,7 @@ MemoryStore <- R6::R6Class("MemoryStore",
#' Read-only. Depends on the `crul` package.
#'
#' @rdname HttpStore
#' @importFrom memoise memoise timeout
#' @export
HttpStore <- R6::R6Class("HttpStore",
inherit = Store,
Expand Down Expand Up @@ -393,9 +394,9 @@ HttpStore <- R6::R6Class("HttpStore",
},
memoize_make_request = function() {
if(private$cache_enabled) {
private$make_request_memoized <- memoise::memoise(
private$make_request_memoized <- memoise(
function(key) private$make_request(key),
~memoise::timeout(private$cache_time_seconds)
~timeout(private$cache_time_seconds)
)
} else {
private$make_request_memoized <- private$make_request
Expand Down
2 changes: 1 addition & 1 deletion R/zarr-array.R
Original file line number Diff line number Diff line change
Expand Up @@ -1295,7 +1295,7 @@ ZarrArray <- R6::R6Class("ZarrArray",

#' S3 method for as.array
#'
#' @param obj object
#' @param x object
#' @param ... not used
#' @keywords internal
#' @export
Expand Down
4 changes: 2 additions & 2 deletions man/as.array.ZarrArray.Rd

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading