Pre-release: v1.3.0a

# Release Notes for pre-release 1.3

Module Restructuring

• Restructured module sections into three components:

  • Core: Central components
  • Operations: Relating specifically to group operations.
  • Phases: Classes for specific phases within the pipeline (scan, compute etc.)
  • Added a Tests module for automated testing.

• Other details

  • The entire module is now referred to as padocc throughout.
  • padocc is consistent with other packages in the CEDA package landscape in terms of its use of poetry for dependency management.

## Filehandlers

• Generic properties of all filehandlers:

  • move_file: Specify a new path/name for the attached file.
  • remove_file: Delete the file from the filesystem.
  • file: Name of the attached file.
  • filename: Full path to the attached file.

• Magic Methods of all filehandlers - implementations vary.

  • __contains__: Check if a value exists within the filehandler contents.
  • __str__: String representation of the specific instance.
  • __repr__: Programmatic representation of the specific instance.
  • __len__: Length of contents within the filehandler (where appropriate)
  • __iter__: Allows generation of iterable content of filehandler (where appropriate)
  • __getitem__: Filehandlers are indexable.

• Other Methods

  • Append: ListFileHandler operates like a list with append() function.
  • Set/Get: Set or get the value attributed to a filehandler
  • Close: Save content of the filehandler to the filesystem.

• Methods for Special Filehandlers

  • add_download_link: Specific to the Kerchunk JSON Filehandler, resets the path for all chunks to use the CEDA dap connection.
  • add_kerchunk_history: Adds specific parameters to the Kerchunk content's history parameter to state when it was generated etc.
  • clear: Clears the Store-type filehandlers of all internal files.
  • open (beta): Open a cloud-format (Store or JSON) Filehandler as a dataset

• Other Features

  • Conf: JSON Filehandlers allow a special property to be supplied, a default dictionary which contains some template values. These are then applied before saving any content, where all new values override the template.
  • Documentation: Added the documentation page for filehandlers and subcomponents.

Project Operator

Documentation is provided for the project operator, detailed here are some of the key features.

• Key Methods

  • info: Obtain information about the specific operator instance.
  • help: Get help with public methods
  • run: Can only be used with a Phase Operator (see below) for running a phased operation. All errors are handled and logged when using this function.
  • increment_versions: Increment major or minor versions

• Properties

  • dir: The directory on the filesystem where all project files can be found
  • groupdir: The path to the group directory on the filesystem.
  • cfa_path: The path to a cfa dataset for this project.
  • outproduct: The complete filename of the output product
  • outpath: The combined path to the output product.
  • revision: The full revision describing the product version i.e 'kr1.0'
  • version_no: The version number (second part of the revision i.e '1.0') with a major and minor component.
  • cloud_format [Editable]: The cloud format being used in current workflows.
  • file_type [Editable]: The file type being used in current workflows.
  • source_format: The source format detected for this project.

## Group Operator

The group operator enables the application of phased operators across a group of datasets at once, either by parallel deployment or serial handling.

• Key Methods

  • info: Obtain useful information about the group
  • help: Find helpful user functions for the group.
  • merge/unmerge (beta): Allows the merging and unmerging of groups of datasets.
  • run: Perform an operation on a subset of full set of the datasets within the group.
  • create_sbatch: Create a job deployment to SLURM for group processing.
  • init_from_file [Mixin]: Initialise a group from parameters in a CSV file.
  • add/remove_project [Mixin]: Add or remove a specific project from a group (Not implemented in pre-release)
  • get_project [Mixin]: Enables retrieval of a specific project, also accomplished by indexing the group which utilises this function.

• Still in development:

  • summary_data: Summarise the data across all projects in the group, including status within the pipeline.
  • progress: Obtain pipeline specific information about the projects in the group.
  • create_allocations: Assemble allocations using binpacking for parallel deployment.

• Magic Methods

  • __str__: String representation of the group
  • __repr__: Programmatic representation of the group.
  • __getitem__: Group is integer indexable to obtain a specific dataset as a ProjectOperator.

## Phased Operators

The phased operators can be used to individually operate on a specific project, although
it is instead suggested that the GroupOperator.run() method is used as this includes all error logging
as part of the project operator. Specifics of the phased operators are described below:

• Scan Operator: Scan a subset of the source files in a specific project and generate some extrapolated results for the whole dataset, including number of chunks, data size and volumes etc.
• Compute Operator: Inherited by DatasetProcessors within the compute module (Kerchunk, Zarr, CFA etc.) and enables the computation step. The scan operator uses the compute processors with a file limiter, to operate on a small subset for scanning purposes.
• Validate Operator: Perform dataset validation using the CFA dataset generated as part of the pipeline. If a CFA dataset is the only dataset generated, this step is currently not utilised.

Future improvements:

• Reorganisation of the compute operator as the current inheritance system is overly complicated.
• Addition of an ingest operator for ingestion into STAC catalogs and the CEDA Archive (CEDA-only.)

Version 1.2 Allocations and Banding

Updates for version 1.2:

• Pipeline to Aggregate Data for Optimised Cloud Capabilities (padocc) - New official name for the pipeline.

Assessor (addition 1.2.a)

• Two new modes added! (match and status_log)
• Added new display options! (allocations and bands now displayed)
• Bug fixes:
  • merge_old_new - issue with indexing different types of lists. (1.2.a1)
  • cleanup - now able to delete allocation directories. (1.2.a2)
  • progress - now able to match multiple error types. (1.2.a3)

Allocation (addition 1.2.b)

• Added allocations for compute processes with estimations using binpacking - requires specific flag to enable.
• Added general purpose bands for rerunning different processes - will look at past runs and add time for failed jobs.
  • Uses default values for time for each phase, unless --allow-band-increase is enabled in which case previous runs are considered.

Documentation (addition 1.2.c)

• Added developer's guide for adding new features!
• Updated flag listings for all tool scripts.

Group Run (addition 1.2.d)

• Added default times for different phases
• Added deployment function for multiple arrays from within a single call! Allocations and bands can now be deployed (no current limit to number of array jobs that can be deployed simultaneously)
• Added pre-deployment input requirement to check deployments are as expected.
• Minor bug fixes
  • Verbose flag now carries over to subprocesses (1.2.d1)

Compute (addition 1.2.e)

• Added Zarr processor!
• Rearranged all processors with new names and class inherritance.
• Added ProjectProcessor parent class from which KerchunkDSProcessor now inherits!
• New functions for checking variable shapes and determining behaviour which helps optimise processes within the pipeline.

Errors (addition 1.2.f)

• Added NaNComparisonError - for consistent issues with comparing arrays (1.2.f1)
• Added RemoteProtocolError - if the remote protocol cannot be handled properly (1.2.f2)
• Added SourceNotFoundError - for resources that failed to open (1.2.f3)
• Added ArchiveConnectionError - catches fsspec ReferenceNotReachable for multiple tries (1.2.f4)
• Added KerchunkDecodeError - issue opening Kerchunk file (normally time decode related) (1.2.f5)
• Added FullsetRequiredError - raised instead of risking a timeout in validation (1.2.f6)

Index_Cat (addition 1.2.g)

• Initialised script for later use pushing Kerchunk records to an index

Ingest (addition 1.2.h)

• Initialised script with some basic functions to use when ingesting data to the CEDA archive, also checks download links have been added properly.

Init (addition 1.2.i)

• Updated docstrings

Logs (addition 1.2.j)

• Added log_status fetch function
• Updated init_logger to ensure filehandler exists.

Scan (addition 1.2.k)

• Removed unused function eval_sizes
• Altered scan setup to use instances of processor classes.
• Added new detail-cfg attributes!
• Added override_type for specifying Zarr as an output type.

Utils (addition 1.2.l)

• Added new switches to BypassSwitch class for fasttrack, skip link addition.
• Reconfigured remote protocol option in open_kerchunk.
• Added function specifically to get the blacklist.
• Added get/set_last_run routines for band increases if jobs time out.
• Added find_divisor and find_closest routines for use in allocations.

Validate (addition 1.2.m)

• Integration of new errors
• Multiple tries of fetching Kerchunk/Xarray data with different options if required.
• Added array flattening at point of checking NaN values, the flattened arrays are then used throughout, and with the new error codes for unreachable chunks, this means once the data is fetched successfully it can be kept and used for all tests.

Notebooks (addition 1.2.n)

• Renamed simple scan notebook.
• Initialised pipeline test notebook.

Single Run (addition 1.2.o)

• Reconfigured how allocations/bands/subsets work for single/multiple processes.
• Reconfigured logger creation when dealing with multiple processes in a single job.
• Added override_type flag for compute phase.

Version 1.1 PPC Error Tracking

New Software Features:

• Per Project Code (PPC) Error tracking
• Individual log files for each dataset, updated automatically with filehandler updates built into the pipeline - for upcoming job allocation improvements.
• Scanning improvements and identification of types of dimensions.
• Support for virtual dimension additions (file_number)
• BypassSwitch option and default changes.

New Documentation:

• Documentation Updates
• Example CCI Water Vapour files and tutorial
• Kerchunk Powerpoints

v1.0.2

Version 1.0.2

• Major documentation overhaul
• Added BypassSwitch for better control of switch options
• Added features to Assessor:

1. Blacklist and proj_code list concatenation with existing files.
2. Slurm error type recognition and better labelling.

Alpha 1.0.1

Version 1.0.1

• errors.py - contains all custom error classes for Kerchunk pipeline
• logs.py - contains logger content and other utils
• setup.py - installs pipeline scripts into environment
• validate.py - thorough validation added (try subset, then try full fileset)

Bug Fixes:

• Dap link in cached files: Pipeline now writes cache files BEFORE concatenation to ensure no linkage.
• Dimension loading issue: Known issue with remote_protocol set - greater distinction between pre-concatenation and post-concatenation Kerchunk files.

Features added:

• Memory flag: Assign specific memory to parallel job arrays
• In-place validation: Pipeline allows rechecking of 'complete' datasets.
• Multi-dimension concatenation support
• Identical_dims identification within compute phase.
• Added custom error classes for edge cases.

Alpha 1.0

First release (alpha) of CEDA Kerchunk Pipeline. Includes Init, Scan, Compute and Validate phases but no Catalog/Ingest control solution. Kerchunk files created by the pipeline are fully verified so all results from NetCDF solutions will be the same as the Kerchunk method.