Updates to config post-processing

[anton-seaice]

Apologies for the multi-topic post, there are some common threads though. There are a few things we can do in config post processing.

Questions:
The scripts are general to all configurations. So are we going to add them individually (create a tools folder in the style of OM2) to each configuration branch? Or should we store them in a central location (e.g. in vk83)? Or load them as a package or submodule? (Or update payu)

The script will add a dependency to need python and access-nri-intake, shall we add these to the vk83 payu module or is there a better way to manage them?

Requirements:

1. Build an intake datastore. This looks approximately like this for a first run (probably needs a dask cluster?) and we can save the datastore at the top level of the in the archive folder :

from access_nri_intake.source import builders

builder = builders.AccessOm3Builder(
    path=...
)

builder.build()

# Log invalid assets:

builder.invalid_assets

# Save the datastore to a file (json)

builder.save(name='intake_datastore', description=..., directory=...)

Subsequent runs will have to remake the whole datastore

2. Concatenate the sea ice daily output. This looks very similar to https://github.com/ACCESS-NRI/om2-scripts/blob/main/concatenate_ice/concat_ice_daily.sh but with some paths adjusted. Should this be a sync or an archive user-script?

3. Convert MOM6 output to netcdf4 and compress. This should be achievable with ncks (at lower resolutions) which is already in payu-env but may interact with mppn-combine at higher resolution (like OM2?).

Ping @aidanheerdegen

[anton-seaice]

I think archive userscripts use the PBS job the model was run on, and sync userscripts use a seperate job (ACCESS-NRI/access-om2-configs#31 (comment))

[anton-seaice]

I drafted something here for points 1 & 2. I will do 3 later

ACCESS-NRI/access-om3-configs@1deg_jra55do_ryf...1deg_jra55do_ryf.iss182

I dont understand how to run multiple archive userscripts (is it possible?). This fails:

userscripts:
    setup: tools/setup_cice_restarts.sh
    archive: 
        - tools/archive_cice_restarts.sh
        - tools/concat_ice_daily.sh
        - python3 tools/build_intake_ds.py

This also fails (it only runs the first one)

userscripts:
    setup: tools/setup_cice_restarts.sh
    archive: tools/archive_cice_restarts.sh & tools/concat_ice_daily.sh & python3 tools/build_intake_ds.py

[anton-seaice]

I dont understand how to run multiple archive userscripts (is it possible?). This fails:
...

This should be possible once payu-org/payu#452 is implemented, which will be in next release

[anton-seaice]

Leaving a note to myself to fix two things in the prototype:

• The ice daily needs concatenating BEFORE building the intake cat (not at the same time)
• This needs re-ordering an extra memory limit var, so the limit is correct per worker:

workers = int(environ["PBS_NCPUS"])
    if workers > 48:
        workers = 48
    client = Client(n_workers=workers , threads_per_worker=1, memory_limit=int(environ["PBS_VMEM"]) / workers)

• Add git hash metadata to the output (make a common script for that?)

[anton-seaice]

I've moved the scripts into COSIMA/om3-scripts#13

[anton-seaice]

Its not possible to do the concat of daily data and generate a datastore until ACCESS-NRI/access-nri-intake-catalog#176 is resolved.

[anton-seaice]

Closed through :

• Updates to 1deg_jra55do_ryf ACCESS-NRI/access-om3-configs#91
• 1deg jra55do iaf.cherrypicks ACCESS-NRI/access-om3-configs#102
• 025deg jra55do ryf.cherrypicks ACCESS-NRI/access-om3-configs#103
• 1deg jra55do ryf.cherrypicks ACCESS-NRI/access-om3-wav-configs#44
• 1deg jra55do iaf.cherrypicks ACCESS-NRI/access-om3-wav-configs#45