MAPL develop is non-zero-diff to main - 2024-Sep-11

[mathomp4]

My nightly tests have informed me that some time late last week, MAPL develop became non-zero-diff to MAPL main (aka MAPL 2.47). I missed this because of other issues I was having with the tests. Fixed those and new issue exposed.

From my testing, the culprit seems to be #3007 . If I build GEOS from commit d888902 from before that PR and compare to MAPL 2.47 it is zero-diff. If I move after that PR (say commit 677ee24), it is non-zero-diff.

[bena-nasa]

This just getting bizarre. I've reproduced, from what I can tell, the same files are being read, seems be producing the same values. Even more bizarre, the model doesn't to go non-zero diff right away; 4 time steps the model is still zero diff, but then after 30 minutes goes non-zero diff. If I /dev/null enough out in the sulfate component it goes zero-diff. But this doesn't make sense because most of the GOCART items have this only update once a day thing (and all the ones I've not commented out are of this variety). So after the first step ExtData won't touch the pointers until and the next day, and I have confirmed with prints!

I'm now going to trying running ExtDataDriver.x in the mode where I read in the same emissions as the model, then write them back out immediately with History and see if main and develop produce the same results

[mathomp4]

Update: I looked at the nightly tests and it looks like it is AMIP GOCART that has the issue and not Ops:

Full Log File: /discover/nobackup/mathomp4/SystemTests/logs/AGCM/2024-09-11/log.MAPL.00:20:05

Runtype         Clone Build Build Time Ctest/Run Run/Comp/ESMF  Regress/LT/SS  MAPL Comp Remap Run/Comp SCM Run/Comp NAS Xfer NAS Comp
-------         ----- ----- ---------- --------- -------------  -------------  --------- -------------- ------------ -------- --------
mapldev         pass  pass  26 min      -- / --  pass/FAIL/pass pass/pass/pass FAIL      pass/pass      pass/FAIL    pass      --
mapldevdebug     --   pass  17 min      -- / --  pass/FAIL/pass  -- / -- / --  FAIL       -- / --        -- / --     pass      --
mapldevrep       --    --    --         -- / --  pass/FAIL/pass pass/pass/pass FAIL       -- / --        -- / --     pass      --
mapldevrepnoinc  --    --    --         -- / --  pass/FAIL/pass pass/pass/pass FAIL       -- / --        -- / --     pass      --
mapldevops181    --    --    --         -- / --  pass/pass/pass pass/pass/pass pass       -- / --        -- / --     pass      --
mapldevmom6      --    --    --         -- / --  pass/FAIL/pass  -- / -- / --  FAIL       -- / --        -- / --     pass      --

The Ops emissions test worked.

[bena-nasa]

I think I figured it out. I think I broke this "flipping" we do in certain cases. It was part of the vertical stuff I said, we don't need cause that's going to be completely redone, well, apparently that was used in some of our current files.

And ExtDataDriver.x proves it's worth again

[tclune]

Good job!

[mathomp4]

So I guess what this means is that the ExtData flipping is something we only do in AMIP emissions? Or maybe we got lucky with how the ops is set up?

GOCART is still a bit of a black box to me :)