Version 1.2 Allocations and Banding

Updates for version 1.2:

• Pipeline to Aggregate Data for Optimised Cloud Capabilities (padocc) - New official name for the pipeline.

Assessor (addition 1.2.a)

• Two new modes added! (match and status_log)
• Added new display options! (allocations and bands now displayed)
• Bug fixes:
  • merge_old_new - issue with indexing different types of lists. (1.2.a1)
  • cleanup - now able to delete allocation directories. (1.2.a2)
  • progress - now able to match multiple error types. (1.2.a3)

Allocation (addition 1.2.b)

• Added allocations for compute processes with estimations using binpacking - requires specific flag to enable.
• Added general purpose bands for rerunning different processes - will look at past runs and add time for failed jobs.
  • Uses default values for time for each phase, unless --allow-band-increase is enabled in which case previous runs are considered.

Documentation (addition 1.2.c)

• Added developer's guide for adding new features!
• Updated flag listings for all tool scripts.

Group Run (addition 1.2.d)

• Added default times for different phases
• Added deployment function for multiple arrays from within a single call! Allocations and bands can now be deployed (no current limit to number of array jobs that can be deployed simultaneously)
• Added pre-deployment input requirement to check deployments are as expected.
• Minor bug fixes
  • Verbose flag now carries over to subprocesses (1.2.d1)

Compute (addition 1.2.e)

• Added Zarr processor!
• Rearranged all processors with new names and class inherritance.
• Added ProjectProcessor parent class from which KerchunkDSProcessor now inherits!
• New functions for checking variable shapes and determining behaviour which helps optimise processes within the pipeline.

Errors (addition 1.2.f)

• Added NaNComparisonError - for consistent issues with comparing arrays (1.2.f1)
• Added RemoteProtocolError - if the remote protocol cannot be handled properly (1.2.f2)
• Added SourceNotFoundError - for resources that failed to open (1.2.f3)
• Added ArchiveConnectionError - catches fsspec ReferenceNotReachable for multiple tries (1.2.f4)
• Added KerchunkDecodeError - issue opening Kerchunk file (normally time decode related) (1.2.f5)
• Added FullsetRequiredError - raised instead of risking a timeout in validation (1.2.f6)

Index_Cat (addition 1.2.g)

• Initialised script for later use pushing Kerchunk records to an index

Ingest (addition 1.2.h)

• Initialised script with some basic functions to use when ingesting data to the CEDA archive, also checks download links have been added properly.

Init (addition 1.2.i)

• Updated docstrings

Logs (addition 1.2.j)

• Added log_status fetch function
• Updated init_logger to ensure filehandler exists.

Scan (addition 1.2.k)

• Removed unused function eval_sizes
• Altered scan setup to use instances of processor classes.
• Added new detail-cfg attributes!
• Added override_type for specifying Zarr as an output type.

Utils (addition 1.2.l)

• Added new switches to BypassSwitch class for fasttrack, skip link addition.
• Reconfigured remote protocol option in open_kerchunk.
• Added function specifically to get the blacklist.
• Added get/set_last_run routines for band increases if jobs time out.
• Added find_divisor and find_closest routines for use in allocations.

Validate (addition 1.2.m)

• Integration of new errors
• Multiple tries of fetching Kerchunk/Xarray data with different options if required.
• Added array flattening at point of checking NaN values, the flattened arrays are then used throughout, and with the new error codes for unreachable chunks, this means once the data is fetched successfully it can be kept and used for all tests.

Notebooks (addition 1.2.n)

• Renamed simple scan notebook.
• Initialised pipeline test notebook.

Single Run (addition 1.2.o)

• Reconfigured how allocations/bands/subsets work for single/multiple processes.
• Reconfigured logger creation when dealing with multiple processes in a single job.
• Added override_type flag for compute phase.