OSX Test and Monitor Test GA tests are failing for all Pythons

[valeriupredoi]

Last night's OSX tests have all died for all Python with a multitude of problems - it seems fiona is at it again, and a lot of FLAKE8 problems. Diffing the envs between passed and failed identical tests, I find these packages have changed:

<     + curl                             7.82.0  h9f20792_0                   conda-forge/osx-64                         149kB
---
>     + curl                             7.81.0  hf45b732_0                   conda-forge/osx-64                         148kB
53c53
<     + expat                             2.4.7  h96cf925_0                   conda-forge/osx-64                         150kB
---
>     + expat                             2.4.6  h96cf925_0                   conda-forge/osx-64                         149kB
61c61
<     + fontconfig                      2.13.96  h676cef8_1                   conda-forge/osx-64                         282kB
---
>     + fontconfig                      2.13.96  h10f422b_0                   conda-forge/osx-64                         297kB
123c123
<     + libcurl                          7.82.0  h9f20792_0                   conda-forge/osx-64                         327kB
---
>     + libcurl                          7.81.0  hf45b732_0                   conda-forge/osx-64                         327kB
216c216
<     + python                           3.10.2  h1dd9edd_4_cpython           conda-forge/osx-64                          13MB
---
>     + python                           3.10.2  h1dd9edd_3_cpython           conda-forge/osx-64                          13MB
219c219
<     + python-stratify               0.2.post0  py310h7f5fb2b_1              conda-forge/osx-64                         161kB
---
>     + python-stratify               0.2.post0  py310h81f86ea_1              conda-forge/osx-64

It is possible the new Python build is causing this (I can't think of any other package from that list that may cause such mayhem). I'll try pin it see if we get anywhere, if not, am just gonna turn off the tests. @zklaus @bvreede you guys have any ideas?

[zklaus]

Not completely sure, but there was a hiccup with fontconfig, see conda-forge/fontconfig-feedstock#53. That would hit us via

flowchart LR
    fontconfig --> poppler[conda-forge/poppler-feedstock#124] --> gdal --> fiona
    fontconfig --> cairo[conda-forge/cairo-feedstock#62] --> poppler

Loading

If that's right, conda-forge/poppler-feedstock#124 should solve the issue. Let's wait and see.

[valeriupredoi]

brilliant, cheers, Klaus, gonna test that in Core! If it is that, we'll just sit tight and wait - a fonts package wreaking so much havoc though?

[valeriupredoi]

@zklaus you were right, it is blithering fontconfig - I ran a check with it pinned at the previous build https://github.com/ESMValGroup/ESMValCore/runs/5481211780?check_suite_focus=true

[zklaus]

Ok, so that seems to have done the trick. Mostly. One issue remains: on osx (and arm64), Python 3.7 is no longer supported by conda-forge (see here). So that version will not be repaired.

Should we:

• Drop 3.7 altogether
• Drop 3.7 for osx
• Attempt some pinning trickery for osx

@ESMValGroup/esmvaltool-coreteam @ESMValGroup/technical-lead-development-team ?

[schlunma]

I would go for option 1 or 2, with a slight preference on option 1 (drop 3.7 altogether). Do we need another release candidate for that?

We are running the recipe tests with python>3.7, so in principle this shouldn't affect our tests.

[zklaus]

My vote also goes to option 1. Another release candidate? Hm. Strictly speaking...

[schlunma]

If we do an rc6 then it would be great to drop 3.7 today and release rc6 today or tomorrow, so that @remi-kazeroni can do the test with that. There is no need to do the tests with rc5.

[zklaus]

@valeriupredoi, any opinion?

[valeriupredoi]

Hang on, on the bus to work, will reply in more detail, but for now 1 and/or 2 should do the trick 🍺 Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Thu, 10 Mar 2022, 10:09 Klaus Zimmermann, ***@***.***> wrote: @valeriupredoi <https://github.com/valeriupredoi>, any opinion? — Reply to this email directly, view it on GitHub <#2584 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG5EFI3DXHR6RR5L42TJMYTU7HC5XANCNFSM5QJPYL7Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[remi-kazeroni]

If we do an rc6 then it would be great to drop 3.7 today and release rc6 today or tomorrow, so that @remi-kazeroni can do the test with that. There is no need to do the tests with rc5.

I leave it up to you. But given the time constraints with Mistral replacement, I think we can only afford one last round of testing. I can use rc5 or rc6, as you tell me.

[schlunma]

I will prepare a PR that drops 3.7, then further discussion is easier.

[schlunma]

See ESMValGroup/ESMValCore#1530 for ESMValCore. If we decide to merge this, then I will open a similar one for ESMValTool shortly after. The one for ESMValCore should have priority for the release.

[valeriupredoi]

ok guys, off the bus, finally - my take was drop 3.7 for OSX now and then drop 3.7 for linux64 (arm64 too, but we don't have it yet anyway) with the next release - but am flexible on it - basically if we drop support for 3.7 altogether now I will not cry too much - cheers for the heads up, Klaus! 👍

[schlunma]

See #2585 for ESMValTool.

As mentioned, would be great to reach a conclusion today, so we can initialize the last round of recipe tests with a new rc.

[bouweandela]

If we do what V says, would that reduce the amount of work because no extra release candidate + testing is needed? Because then i would recommend that.

[schlunma]

rc5 is not on conda-forge yet, so it's not really less amount of work if we merge #1530 this afternoon and do a rc6 immediately after (and it's also cleaner as then rc6=2.5.0 most likely). I would actually prefer to drop 3.7 altogether now.

[zklaus]

Yes, let's drop 3.7 altogether. Otherwise, we would also have to change the conda-forge package from noarch to arch-specific packages.

[valeriupredoi]

ah yes, that is a very powerful point @zklaus - nuke it! Actually, poor choice of words on my part 😆