chore: bump pyspark version

[project-defiant]

✨ Context

This PR closes #3189
This PR closes #3680
and links to opentargets/orchestration#94

🛠 What does this PR implement

This PR focuses on allowing pyspark>=3.5.0, <3.6 along with newest hail version, that supports it.
The changes above caused trouble (errors) for poetry to resolve the dependencies. This lead to more developments, the full list is described below:

• replaced poetry with uv as a dependency management tool (~5x faster dependency resolution without issues)
• added support for X.Y.Z-devXX releases on push to dev branch
• updated way to create dev dataproc cluster (Open Targets users only) - see details below
• add support for multiple python versions in gentropy (py310, py311, py312) py313 support failure due to missing BLAS when installing scipy from wheel
• support for pyspark>=3.5.0, <3.6
• bump hail version to sync with pyspark dependency
• dropped google as dependency, added more granular google-cloud-storage (To be removed in future release with other google dependent packages)
• github actions update to use uv and test for dependency matrix of python versions (requires changes in the repository rules)

dev cluster setup

Previously dev cluster was created from the package pushed to the google cloud storage bucket under the namespace that matched the git ref name. This has now changed, as cluster now install gentropy directly from the github branch (or tag) specified in the orchestration - see changes in opentargets/orchestration#94.

Benefits of new solution

• keeping just one copy of the static files used for running gentropy steps(install_dependencies_on_cluster.sh and cli.py) in the GCS.
• spped up the development process, since the one does not need to wait for the github actions to finish, before they can set up the cluster from airflow on the branch.
• speed up the dependency resolution with uv pip install (pip resolution of gentropy with support for multiple python versions takes ~ 20minutes leading to initialization actions timeout, uv does the same thing with around 1.5min).

🙈 Missing

🚦 Before submitting

• Do these changes cover one single feature (one change at a time)?
• Did you read the contributor guideline?
• Did you make sure to update the documentation with your changes?
• Did you make sure there is no commented out code in this PR?
• Did you follow conventional commits standards in PR title and commit messages?
• Did you make sure the branch is up-to-date with the dev branch?
• Did you write any new necessary tests?
• Did you make sure the changes pass local tests (make test)?
• Did you make sure the changes pass pre-commit rules (e.g poetry run pre-commit run --all-files)?

[ireneisdoomed]

Thank you!
2 questions:

1. Should we also give support to pyspark versions lower than 3.5? Afaik, there is nothing in the API that is not compatible. I know we have custom code to compare dataframes, something that Pyspark 3.5 supports
2. We need to update the Dataproc version too. Should we make a pipeline run to test nothing breaks?

[project-defiant]

Thank you! 2 questions:

1. Should we also give support to pyspark versions lower than 3.5? Afaik, there is nothing in the API that is not compatible. I know we have custom code to compare dataframes, something that Pyspark 3.5 supports
2. We need to update the Dataproc version too. Should we make a pipeline run to test nothing breaks?

These are very nice points!

1. I am aftraid, that poetry will not allow us to supoport more pyspark versions, latest hail requires version between 3.5 to 3.6, otherwise we could do it.
2. Yes I am planning to do that, just right after I finish the harmonisation :). Let's wait for this check before we merge.

[ireneisdoomed]

Really cool!! New PySpark, support for several Python versions (dev version set to 3.11), and migration to UV.

I suggest changing the PR title, this is clearly a lie haha

Setting up the dev environment was smooth simply by doing make setup-dev (except that uv wasnt in my path, see minor comment below).

I haven't used uv myself, so I can't really evaluate pros and cons. M only questions is to confirm that we are not missing anything from CI/CD after the migration?

[ireneisdoomed]

Nice, I didn't know about --porcerlain :D

[ireneisdoomed]

🥳

[ireneisdoomed]

The Gentropy image will be automatically updated, right?

[ireneisdoomed]

After installing the environment, uv wasn't on my path. Should we add export PATH="$HOME/.local/bin:$PATH" here?

[ireneisdoomed]

Please test this yourself, but the current command goes idle without running any tests

Suggested change

	@uv run pytest
	@uv run pytest .