Mesh nonexperimental #6061

pp-mo · 2024-07-18T16:02:25Z

Addresses #6057

WIP will need some rebase when outstanding changes are merged :

Also remaining

tidying to-do :

re-integrate various 'extra' code elements back into legacy sourcefiles
- ugrid.cf --> iris.fileformats.cf
- ugrid.load --> iris.fileformats.netcdf.loader
- ugrid.save --> iris.fileformats.netcdf.saver
- ugrid.metadata--> iris.common.metadata
that leaves ugrid.mesh and ugrid.utils in place,
- which we should probably rename as iris.mesh and iris.mesh.utils

For simplicity, I'm going to defer that to another PR : see #6073

codecov · 2024-07-18T16:12:13Z

Codecov Report

Attention: Patch coverage is 66.66667% with 16 lines in your changes missing coverage. Please review.

Project coverage is 89.69%. Comparing base (304c5e0) to head (f8b7644).
Report is 1 commits behind head on main.

Files	Patch %	Lines
lib/iris/experimental/ugrid.py	0.00%	16 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6061      +/-   ##
==========================================
- Coverage   89.75%   89.69%   -0.07%     
==========================================
  Files          90       91       +1     
  Lines       22981    22997      +16     
  Branches     5026     5026              
==========================================
  Hits        20627    20627              
- Misses       1624     1640      +16     
  Partials      730      730

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

pp-mo · 2024-07-19T14:32:15Z

Status update 2024-07-19 ~15:30

lots of conflicts with main now.
above outstanding PRs to merge, will presumably create more, so holding off for now.

pp-mo · 2024-07-19T18:00:52Z

Status update 2024-07-19 ~1900

rebased + resolved conflicts
needs to be reconciled with #6054 which is still outstanding

Status 2024-07-22

now up-to-date with latest main, and passing.
Still items to complete, as listed in header above. Deciding whether to split these subsequent changes into a separate PR for easier review.

pp-mo · 2024-07-22T16:52:59Z

Status 2024-07-22 ~17:30

Found that benchmark code also needs updating to new module path AND removal of PARSE_UGRID_ON_LOAD
Converted back to draft

What's left to do

Reintegrate code which was separated for optional ugrid #6073 As explained at top, the original brief says to re-integrate all the bits of code which delivered the switchable UGRID support
- probably do in separate PR
- (and one more thing) this PR leaves code full of placeholders like:
  "TODO: rationalise UGRID/mesh handling once iris.ugrid is folded"
  "TODO: complete iris.ugrid replacement"
  so, remove all those
but also, as just discovered, the benchmark code was missed out in Enable UGRID loading always; deprecate PARSE_UGRID_ON_LOAD. #6054, and also from the patch changes here
- probably do that here

pp-mo · 2024-07-23T12:21:11Z

?ready for review?

This roughly hangs together + worth reviewing now.

Though notably, the benchmarks need updating, and I have plans to refactor + (further) rename the API.
(iris.ugrid --> iris.mesh; review what parts are public at top level, and what should be demoted to sub-modules)
Follow-on work is currently being prototyped in #6077

pp-mo · 2024-07-23T17:11:02Z

Hmm benchmarks show a worrying thing..
#6080 (comment)

It's not a fluke, I only did this because of #6075
As it seems to scale, we definitely need to investigate.

bjlittle

@pp-mo Awesome effort 👍

Just a few TODO comments as reminders to address afterwards 👍

As an aside, I noticed that we're pulling the use of the logger across from iris.experimental.ugrid into the iris core. Not sure about our current appetite on the use of logging ATM, but seems like tech debt to me, but that's an out of scope discussion.

Also, with regards to API, rebranding iris.ugrid as iris.mesh is fine from my perspective to be more future UGRID agnostic. However, that means we'd have iris.mesh.mesh which is less than ideal.

In this case, I'd recommend incorporating the iris.ugrid.mesh into __init__.py of iris.ugrid and then rebrand 👍

Otherwise, all good, but there is an iris.tests.stock.__init__ comment to check and service 👍

lib/iris/common/metadata.py

lib/iris/coords.py

lib/iris/experimental/ugrid.py

lib/iris/fileformats/cf.py

lib/iris/tests/unit/ugrid/cf/test_CFUGridMeshVariable.py

lib/iris/tests/unit/ugrid/cf/test_CFUGridReader.py

lib/iris/tests/unit/ugrid/metadata/__init__.py

lib/iris/tests/unit/ugrid/utils/__init__.py

lib/iris/ugrid/load.py

pp-mo · 2024-07-25T10:43:21Z

OK finally I pushed, think I have covered those changes.
I intend to fix all the "carried over" TODOs in a followup PR, and also add a whats-new entry.

@bjlittle please re-review !

Replace experiment.ugrid, including docstrings and imports. Fix test_ParseUgridOnLoad Fix ugrid.load. Remove PARSE_UGRID from t/i/ugrid/test_ugrid_save Remove PARSE_UGRID from t/u/ff/nc/saver/test_save Remove PARSE_UGRID from t/i/exp/geovista/(both) Remove PARSE_UGRID from t/u/tests/stock/test_netcdf

Include alternative index workaround.

for more information, see https://pre-commit.ci

pp-mo · 2024-07-25T11:30:34Z

Thanks so much @bjlittle

And the good news ... the next one might be worse !
(cos moving+renaming stuff is still so hard to manage)

bjlittle · 2024-07-25T11:36:29Z

And the good news ... the next one might be worse !

@pp-mo Bring it! You da man 😄

* Move all iris.experimental.ugrid to iris.ugrid. Replace experiment.ugrid, including docstrings and imports. Fix test_ParseUgridOnLoad Fix ugrid.load. Remove PARSE_UGRID from t/i/ugrid/test_ugrid_save Remove PARSE_UGRID from t/u/ff/nc/saver/test_save Remove PARSE_UGRID from t/i/exp/geovista/(both) Remove PARSE_UGRID from t/u/tests/stock/test_netcdf * Fix type of mesh in iris.tests.stock * Fix Mesh -> MeshXY in experimental.ugrid * Remove PARSE_UGRID_ON_LOAD and ParseUGridOnLoad, leave in iris.experimental.ugrid only. missed * Replace old-style mesh api in benchmarks. * Move misplaced test + fix obsolete PARSE_UGRID usage. * Minimal doctest fixes * Remove obsolete PARSE_UGRID in iris.ugrid * Move all ugrid.metadata into common.metadata. Removed obsolete tests.unit.metadata * Rename iris.ugrid to iris.mesh fix Fix iris.mesh import in tests/stock/__init__ * Move all iris.mesh.save into iris.fileformats.netcdf.save Fix experimental.ugrid import of save_mesh * Fix docs for iris.ugrid -> iris.mesh * Rename iris.mesh.mesh as iris.mesh.components * Tidy imports in experimental.ugrid * Rebrand so mesh.MeshXY is presented as experimental.ugrid.Mesh. * Move ugrid loading support code from iris.mesh to iris.netcdf, and its tests likewise * Fix circular import. * Move mesh.cf code into fileformats.cf, and tests likewise Fix imports in cf ugrid tests. * Reinclude mesh-load functions in iris.mesh; break circular imports * Small fixes to mesh documentation * Added whatsnew * Complete+remove remaining 'TODO's from #6061 * Odd docstring and comment corrections.

* main: Mesh nonexperimental extra (SciTools#6077) Mesh nonexperimental (SciTools#6061) # Conflicts: # lib/iris/experimental/ugrid/cf.py

* upstream/main: Load performance improvement (ignoring UGRID) (SciTools#6088) Adding monthly and yearly arguments to `guess_bounds` (SciTools#6090) Mesh nonexperimental extra (SciTools#6077) Mesh nonexperimental (SciTools#6061) Bump scitools/workflows from 2024.07.4 to 2024.07.5 (SciTools#6076) Updated environment lockfiles (SciTools#6068) Enable UGRID loading always; deprecate PARSE_UGRID_ON_LOAD. (SciTools#6054) Bump scitools/workflows from 2024.07.3 to 2024.07.4 (SciTools#6071)

pp-mo force-pushed the mesh_nonexperimental branch from 4707612 to c545b62 Compare July 18, 2024 17:56

pp-mo linked an issue Jul 19, 2024 that may be closed by this pull request

remove ugrid from experimental #6057

Closed

pp-mo force-pushed the mesh_nonexperimental branch from c545b62 to 5b4e1f3 Compare July 19, 2024 17:59

pp-mo force-pushed the mesh_nonexperimental branch 4 times, most recently from 99308cd to 15d4f84 Compare July 22, 2024 14:53

pp-mo mentioned this pull request Jul 22, 2024

Reintegrate code which was separated for optional ugrid #6073

Closed

pp-mo marked this pull request as ready for review July 22, 2024 16:12

pp-mo marked this pull request as draft July 22, 2024 16:52

bjlittle self-assigned this Jul 23, 2024

bjlittle self-requested a review July 23, 2024 09:14

This was referenced Jul 23, 2024

Mesh nonexperimental extra #6077

Merged

Update UserGuide "Mesh Support" section #6078

Closed

pp-mo marked this pull request as ready for review July 23, 2024 12:18

pp-mo mentioned this pull request Jul 24, 2024

investigate netcdf load slowdown #6081

Closed

bjlittle requested changes Jul 24, 2024

View reviewed changes

bjlittle approved these changes Jul 25, 2024

View reviewed changes

bjlittle enabled auto-merge (squash) July 25, 2024 11:19

pp-mo added 4 commits July 25, 2024 12:19

Adjust references in docs.

01a7aae

Fix type.

2908cfa

Workaround docs build problem.

f28af3e

Include alternative index workaround.

pre-commit-ci bot and others added 3 commits July 25, 2024 12:19

[pre-commit.ci] auto fixes from pre-commit.com hooks

d8e3f48

for more information, see https://pre-commit.ci

Fix Mesh -> MeshXY in experimental.ugrid

d1a2dc5

Import annotations in new files.

2d614b8

bjlittle force-pushed the mesh_nonexperimental branch from f8b7644 to 2d614b8 Compare July 25, 2024 11:20

bjlittle merged commit 7c87fb2 into SciTools:main Jul 25, 2024
20 checks passed

github-actions bot mentioned this pull request Jul 25, 2024

Performance Shift(s): 7c87fb2a #6087

Closed

pp-mo added a commit to pp-mo/iris that referenced this pull request Jul 25, 2024

Complete+remove remaining 'TODO's from SciTools#6061

32c264e

pp-mo mentioned this pull request Jul 26, 2024

Bmfixed 6061 #6089

Closed

stephenworsley added a commit to stephenworsley/iris that referenced this pull request Jul 26, 2024

Merge branch 'main' into ugrid_load_speed

4f9dc90

* main: Mesh nonexperimental extra (SciTools#6077) Mesh nonexperimental (SciTools#6061) # Conflicts: # lib/iris/experimental/ugrid/cf.py

pp-mo mentioned this pull request Jul 26, 2024

Add mesh metas to the iris.common.metadata public API. #6093

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mesh nonexperimental #6061

Mesh nonexperimental #6061

pp-mo commented Jul 18, 2024 •

edited

Loading

codecov bot commented Jul 18, 2024 •

edited

Loading

pp-mo commented Jul 19, 2024

pp-mo commented Jul 19, 2024 •

edited

Loading

pp-mo commented Jul 22, 2024 •

edited

Loading

pp-mo commented Jul 23, 2024 •

edited

Loading

pp-mo commented Jul 23, 2024 •

edited

Loading

bjlittle left a comment

pp-mo commented Jul 25, 2024

pp-mo commented Jul 25, 2024

bjlittle commented Jul 25, 2024

Mesh nonexperimental #6061

Mesh nonexperimental #6061

Conversation

pp-mo commented Jul 18, 2024 • edited Loading

Also remaining

codecov bot commented Jul 18, 2024 • edited Loading

Codecov Report

pp-mo commented Jul 19, 2024

Status update 2024-07-19 ~15:30

pp-mo commented Jul 19, 2024 • edited Loading

Status update 2024-07-19 ~1900

Status 2024-07-22

pp-mo commented Jul 22, 2024 • edited Loading

Status 2024-07-22 ~17:30

What's left to do

pp-mo commented Jul 23, 2024 • edited Loading

?ready for review?

pp-mo commented Jul 23, 2024 • edited Loading

bjlittle left a comment

Choose a reason for hiding this comment

pp-mo commented Jul 25, 2024

pp-mo commented Jul 25, 2024

bjlittle commented Jul 25, 2024

pp-mo commented Jul 18, 2024 •

edited

Loading

codecov bot commented Jul 18, 2024 •

edited

Loading

pp-mo commented Jul 19, 2024 •

edited

Loading

pp-mo commented Jul 22, 2024 •

edited

Loading

pp-mo commented Jul 23, 2024 •

edited

Loading

pp-mo commented Jul 23, 2024 •

edited

Loading