Make the the Python package compatible with Athena PySpark #33
Conversation
| - python: "3.9" | ||
| pandas: "1.4.*" | ||
| numpy: "1.23.*" | ||
| - python: "3.10" | ||
| pandas: "2.2.*" | ||
| numpy: "2.1.*" | ||
| - python: "3.11" | ||
| pandas: "2.2.*" | ||
| numpy: "2.1.*" | ||
| - python: "3.12" | ||
| pandas: "2.2.*" | ||
| numpy: "2.1.*" | ||
| - python: "3.13" | ||
| pandas: "2.2.*" | ||
| numpy: "2.1.*" |
There was a problem hiding this comment.
This matrix feels a little bit clunky to me, given that we only care about specifying the pandas/numpy versions for Python 3.9, but I couldn't think of a clearer way of specifying it. Any ideas?
python/pyproject.toml
Outdated
| "pandas>=1.4.0,<=2.3.0", | ||
| "numpy>=1.23.0,<=2.2.0" |
There was a problem hiding this comment.
The upper bounds are only present here because we're not testing anything higher in the pytest-coverage workflow, given the weird matrix in the workflow. If we can think of a way to remove the hardcoded upper bounds from the workflow matrix, we could remove the upper bound here. We could also just remove the upper bound from this config and accept that we're potentially not testing the latest versions in the pytest-coverage workflow.
python/pyproject.toml
Outdated
| "mypy", | ||
| "pytest", | ||
| "ruff", | ||
| ] | ||
| docs = [ | ||
| "Sphinx>=8.1.3", | ||
| "myst-parser>=4.0.0", | ||
| "pydata-sphinx-theme>=0.16.0", | ||
| "sphinx-pyproject>=0.3.0" | ||
| "Sphinx", | ||
| "myst-parser", | ||
| "pydata-sphinx-theme", | ||
| "sphinx-pyproject", | ||
| "sphinx-autobuild" |
There was a problem hiding this comment.
Since these dependencies are only for development and docs, I think we can afford to be less stringent with the requirements. By removing the version strings, uv will just install the latest version of each package that's compatible with whatever Python/pandas/numpy versions are installed.
There was a problem hiding this comment.
@jeancochrane Your PR here inspired me to look for a more flexible way to do this testing, and I think I cracked it in ccao-data/assesspy#28. Can you take a look at that PR and see if it makes sense to apply here before I do a full review?
jeancochrane
force-pushed
the
jeancochrane/make-python-package-spark-compatible
branch
4 times, most recently
from
ce05828 to
e9f5833
Compare
jeancochrane
force-pushed
the
jeancochrane/make-python-package-spark-compatible
branch
from
e9f5833 to
e34b6d7
Compare
|
Thanks for the great tox example @dfsnow! This PR should be configured to make use of it now, do you mind taking another look:? |
| [tox] | ||
| min_version = 4.0 | ||
| envlist = | ||
| py{39, 310, 311}-lowest |
There was a problem hiding this comment.
We end the lowest-direct tests at Python 3.11 since myst-parser didn't add support for 3.12 until 3.0.
|
OK @dfsnow, ready for another look! |
jeancochrane
merged commit ca15900
into
jeancochrane/further-python-package-migration
jeancochrane
deleted the
jeancochrane/make-python-package-spark-compatible
branch
The Python package we introduced in #31 is not currently compatible with Athena PySpark since it depends on newer versions of Python and pandas that are not available in Athena's list of preinstalled packages. This PR updates the Python package to support the following lower bounds for tools that Athena PySpark includes in its environment:
We also update the
pytest-coverageGitHub workflow to ensure that we're testing the lower bounds of the package's dependencies as well.