Fix type hints for Python versions < 3.10

[bouweandela]

Description

Python 3.8 and 3.9 do not support

a = str | int

Python 3.8 does not support

a = list[str]

even with from __future__ import annotations.

In function signatures, the syntax is supported though.

Credits to @schlunma for figuring this out.

This pull request updates the code so the older Python versions are type hinted correctly.

---

Before you get started

• ☝ Create an issue to discuss what you are going to do

Checklist

It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

• 🧪 The new functionality is relevant and scientifically sound
• 🛠 This pull request has a descriptive title and labels
• 🛠 Code is written according to the code quality guidelines
• 🛠 Changes are backward compatible
• 🛠 The list of authors is up to date
• 🛠 All checks below this pull request were successful

---

To help with the number pull requests:

• 🙏 We kindly ask you to review two other open pull requests in this repository

[codecov]

Codecov Report

Merging #1897 (3c3decd) into main (69a284d) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1897   +/-   ##
=======================================
  Coverage   92.03%   92.03%           
=======================================
  Files         234      234           
  Lines       12034    12034           
=======================================
  Hits        11075    11075           
  Misses        959      959           

Impacted Files	Coverage Δ
esmvalcore/_recipe/from_datasets.py	100.00% <100.00%> (ø)
esmvalcore/dataset.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[valeriupredoi]

not sure this can be solved easily @bouweandela - have a look at this fastapi/typer#348 - am trying to pin typer see what's out of that, but until they fix it upstream I reckon we can't do much - here's the PR where I pin typer to no avail #1900

[bouweandela]

I'm not sure if typer is related to this issue. What puzzles me is that dask uses a very similar configuration, so why is it not working for us?

[valeriupredoi]

I'm not sure if typer is related to this issue. What puzzles me is that dask uses a very similar configuration, so why is it not working for us?

I thought typer was needed if we wanted to mess about with these type hints, I was wrong and I closed that PR - I am wondering though, maybe dask is using a different configuration of standards, or no mypy? Lemme have a look too

[schlunma]

I don't understand how pinning the Python version of mypy is supposed to solve this. Code like

MyType = list[int] | dict

is simply not valid code in Python<3.10 and will always raise these errors, no mypy involved whatsoever.

[valeriupredoi]

that's why one needs to import _future shtuff, Manu. But, looking at dask's setup.cfg - the mypy section, it looks like they are setting it to be for python=3.9 (like Bouwe did here), but with a whole lot more bells and whistles in the opts section - some stuff that I don't even know what they do

[schlunma]

But there is no __future__ stuff in this PR, is there? On the other hand, dask uses that:

https://github.com/dask/dask/blob/81f771e05f57cab2838534c319a9b81f6e5a00cd/dask/highlevelgraph.py#L1

[valeriupredoi]

But there is no __future__ stuff in this PR, is there? On the other hand, dask uses that:

https://github.com/dask/dask/blob/81f771e05f57cab2838534c319a9b81f6e5a00cd/dask/highlevelgraph.py#L1

not in this PR, Manu, since this a fixer PR - Bouwe put those _futures shtuff already, look at esmvalcore/dataset.py or typing.py

[schlunma]

Ah, yes, that makes sense. Sorry, I didn't check that.

I think I know what the problem is: from __future__ import annotations allows the usage of standard types in variable and function definitions , i.e., something like:

def f(x: list[int]) -> dict[str, float]:
    pass

is allowed and works with Python >= 3.7 when using from __future__ import annotations.

Example in our code:

ESMValCore/esmvalcore/_recipe/from_datasets.py

Line 26 in 25e508a

diagnostics: dict[str, dict[str, Any]] = {}

However, as far as I can tell, it is not possible to use this feature in code that is always executed (e.g., to define custom types) in Python < 3.10 like we do here:

ESMValCore/esmvalcore/_recipe/from_datasets.py

Lines 20 to 21 in 25e508a

	Recipe = dict[str, Any]
	Facets = dict[str, Any]

If you look at the error message we get, it's always those two lines. Other code that uses syntax like dict[...] in function and variable definitions does not raise errors.

[valeriupredoi]

Ah, yes, that makes sense. Sorry, I didn't check that.

I think I know what the problem is: from __future__ import annotations allows the usage of standard types in variable and function definitions , i.e., something like:

def f(x: list[int]) -> dict[str, float]:
    pass

is allowed and works with Python >= 3.7 when using from __future__ import annotations.

Example in our code:

ESMValCore/esmvalcore/_recipe/from_datasets.py

Line 26 in 25e508a

diagnostics: dict[str, dict[str, Any]] = {}

However, as far as I can tell, it is not possible to use this feature in code that is always executed (e.g., to define custom types) in Python < 3.10 like we do here:

ESMValCore/esmvalcore/_recipe/from_datasets.py

Lines 20 to 21 in 25e508a

	Recipe = dict[str, Any]
	Facets = dict[str, Any]

If you look at the error message we get, it's always those two lines. Other code that uses syntax like dict[...] in function and variable definitions does not raise errors.

@schlunma I think that's a bingo! Since it doesn't seem to be mypy's fault here - let me see if we get the same with a lower mypy pin, if that's the case then we got us a runner

[schlunma]

TLDR; replace syntax like

• MyType = A | B with MyType = Union[A, B] (e.g., used here)
• MyType = dict[str, Any] with MyType = Dict[str, Any] (e.g., used here)

after importing from typing import Union, Dict.

[valeriupredoi]

yeah I reckon that's it - just figured it out it's not mypy that's at fault here, unless it's a dormant bug that's not been checked since a few versions ago - @schlunma you wanna make the changes you suggest directly in here and let's have us a test? I don't think @bouweandela would mind, since I've already hacked his PR quite a bit, before getting back to its original state 😁

[valeriupredoi]

@schlunma biutiful 😍

[valeriupredoi]

boom! Looking great, cheers, Manu! @bouweandela I took the liberty make this RfR and approve it while I'm at it too

[valeriupredoi]

@bouweandela got a bit of time to look at this, buds?