CI: Utilize uv lockfile for reproducible test environments #6640

danielhollas · 2024-11-25T23:43:05Z

This PR removes the requirement files in requirements/ folder and replaces them with a single universal lockfile uv.lock. "universal" in this case means that it contains resolved package versions for all supported python versions and operating systems. More on the uv lockfiles can be found in their docs: https://docs.astral.sh/uv/concepts/projects/layout/#the-lockfile

When modifying the dependencies in pyproject.toml, the lockfile needs to be updated by running

uv lock

(Note that using e.g. uv add, uv remove or uv sync updates the lockfile automatically)

This relative simplicity compared to the previous custom solution allows for much easier development and I could delete a lot of custom YAML in CI. To check whether the lockfile is up to date, one can simply run

uv lock --locked

This command is run automatically as a pre-commit hook whenever pyproject.toml is changed.

Note

Developers don't necessarily need to use the lockfile locally, nor are they required to have uv installed, unless they need to modify dependencies in pyproject.toml

Closes #6613

In this implementation, the hook will only check whether the lockfile is compatible with pyproject.toml, it will not try to automatically update it, since that should be decided by the developer.

danielhollas · 2024-11-28T20:38:15Z

.github/actions/install-aiida-core/action.yml

+  - name: Install dependencies from uv lock
+    # NOTE: We're also asserting that the lockfile is up to date
+    if: ${{ inputs.from-lock == 'true' }}
+    run: uv sync --locked --extra pre-commit


NOTE: By default we're installing with pre-commit optional dependencies (extras), which transitively includes other extras, see pyproject.toml I think this is the best solution to make things work for now.

Maybe we add the extra all that includes all extra packages to make it more explicit that we check all of them? I am not sure if there is a scenario where we will change something that is not included in pre-commit, but it is also not super clear that we choose pre-commit here to include all extras. At least I would add a comment

I added a comment for now. I agree this is not ideal, I have some ideas but want to keep them for a separate PR since there's a lot going on here already.

Note that if we wanted to always install all the extras, we could just pass the --all-extras parameter to uv. But that would (among others) install the whole jupyter stack which seems wasteful.

.github/CODEOWNERS

danielhollas · 2024-11-28T20:40:52Z

.github/CODEOWNERS

 utils/dependency_management.py  @aiidateam/dependency-manager
-.github/workflows/dm.yml        @aiidateam/dependency-manager


This file does not exist.

danielhollas · 2024-11-28T21:04:45Z

@agoscinski @unkcpz I think this PR is mostly ready now, let me know what you think!

The only thing missing is developer documentation: I would suggest moving the Dependency Management Wiki article to the repo in a separate PR.

codecov · 2024-11-28T21:06:09Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.93%. Comparing base (ef60b66) to head (e62a2ef).
Report is 154 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6640      +/-   ##
==========================================
+ Coverage   77.51%   77.93%   +0.43%     
==========================================
  Files         560      563       +3     
  Lines       41444    41662     +218     
==========================================
+ Hits        32120    32464     +344     
+ Misses       9324     9198     -126

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

unkcpz · 2024-11-29T07:56:55Z

The only thing missing is developer documentation: I would suggest moving the Dependency Management Wiki article to the repo in a separate PR.

Sure, go ahead with a separate PR, thanks!

khsrali · 2024-11-29T13:42:42Z

@danielhollas nice, thanks a lot!
suppose I have an entry in my pyproject.toml as such:
'plumpy@git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy',

would this be understood and accepted by uv lock?

danielhollas · 2024-11-29T14:48:35Z

@khsrali I tried it and it works without a problem. :-)
btw: I don't think you need the #egg=plumpy part at the and of the URL, and uv strips it anyways.

unkcpz

unless they need to modify dependencies in pyproject.toml

I don't understand how this will work. I think if we run uv lock from different machine, the lock file is generated using the environment where the command is run. Which means two people will have two different lock file generated. Is this leading to large number of lines change?

unkcpz · 2024-12-01T23:34:55Z

.github/workflows/ci-code.yml

@@ -96,15 +64,15 @@ jobs:
        python-version: ${{ matrix.python-version }}

    - name: Setup environment
-      run: .github/workflows/setup.sh
+      run: source .venv/bin/activate && .github/workflows/setup.sh


Where is this .venv created from?

It is created by uv sync automatically in the install-aiida-core action above.

can we write it as comment or like this?

Suggested change

run: source .venv/bin/activate && .github/workflows/setup.sh

run: uv venv && source .venv/bin/activate && .github/workflows/setup.sh

I wrote a comment about this both here and in actions/install-aiida-core/action.yml. See 7959c4d

danielhollas · 2024-12-02T00:59:40Z

I think if we run uv lock from different machine, the lock file is generated using the environment where the command is run.

uv should generate a universal lock file independent of your current python version or OS. At least that is my understanding, would be good if you can verify. I have generated the lock here on Linux and Python 3.12

Also you can try using the lock file I generated by running uv sync. It should work regardless of your python version.

danielhollas · 2024-12-02T12:14:51Z

When merging main into this branch, the lock needed to be updated. Just to illustrate, here's how the failed pre-commit hook looked`

❯ pre-commit run -a check-uv-lock
Check uv lockfile up to date.............................................Failed
- hook id: check-uv-lock
- exit code: 2

Resolved 254 packages in 126ms
error: The lockfile at `uv.lock` needs to be updated, but `--locked` was provided. To update the lockfile, run `uv lock`.

Then all I needed to do is to run uv lock

❯ uv lock
Resolved 254 packages in 152ms
Added execnet v2.1.1
Added pytest-xdist v3.6.1

Note, if we wanted, we can also update the lock automatically as part of the pre-commit hook to avoid the extra manual step?

danielhollas · 2024-12-02T12:18:00Z

I was thinking about similar thing with pip-compile,

Yeah, the previous solution using requirements.txt files was done with pip-compile, but as you pointed out, the major downside is that pip-compile does depend on the python version that you use to run it, so to properly update all the requirement files you'd need to install python versions 3.9-3.12.

unkcpz · 2024-12-02T12:43:23Z

Note, if we wanted, we can also update the lock automatically as part of the pre-commit hook to avoid the extra manual step?

Yes, I think this is better to avoid manual run of the pre-commit locally.

agoscinski

Thanks for the work @danielhollas I think this is very useful. I needed some time to wrap my head around the changes. I am trying it out now in #6600

.github/workflows/test-install.yml

agoscinski · 2024-12-03T17:25:02Z

I had an issue with scipy, needed to be 1.14.1 for the python 3.13 but uv lock and uv lock --upgrade did not upgrade even though my local version was 1.14.1, then with uv lock --upgrade-package scipy==1.14.1 it finally told me that it does not upgrade because scipy==1.14.1 does not support python 3.9. So I needed to specify the scipy version for the python 3.13 separately. I would say it worked very well.

agoscinski

Just some minor changes to add doc here and there. Thanks for the work! I think minor changes + a rebase and then we can merge

agoscinski · 2024-12-03T17:45:58Z

.github/actions/install-aiida-core/action.yml

+  - name: Install dependencies from uv lock
+    # NOTE: We're also asserting that the lockfile is up to date
+    if: ${{ inputs.from-lock == 'true' }}
+    run: uv sync --locked --extra pre-commit


Maybe we add the extra all that includes all extra packages to make it more explicit that we check all of them? I am not sure if there is a scenario where we will change something that is not included in pre-commit, but it is also not super clear that we choose pre-commit here to include all extras. At least I would add a comment

.github/actions/install-aiida-core/action.yml

agoscinski · 2024-12-03T17:51:00Z

.github/workflows/ci-code.yml

@@ -96,15 +64,15 @@ jobs:
        python-version: ${{ matrix.python-version }}

    - name: Setup environment
-      run: .github/workflows/setup.sh
+      run: source .venv/bin/activate && .github/workflows/setup.sh


can we write it as comment or like this?

Suggested change

run: source .venv/bin/activate && .github/workflows/setup.sh

run: uv venv && source .venv/bin/activate && .github/workflows/setup.sh

agoscinski · 2024-12-03T17:58:27Z

.github/workflows/test-install.yml

-        issue-number: ${{ github.event.number }}
-        body: |
-          Please update the requirements/ files to ensure that they
-          are consistent with the dependencies specified in the 'pyproject.toml' file.


I feel bad deleting this, but maintaining this seems to me me not worth the effort. Especially, since we don't update dependencies frequently.

Why do you feel bad? None of this is needed now when we have the lockfile, no?

danielhollas · 2024-12-04T12:15:18Z

@agoscinski thanks for the review! I've merged in main and added more comments, should be good to go.

There are definitely things that could be improved, but I'll save that for a followup.

khsrali · 2024-12-04T13:10:26Z

@danielhollas , stupid question:
do we need to list uv as dependency in pyproject.toml?

danielhollas · 2024-12-04T13:37:25Z

do we need to list uv as dependency in pyproject.toml?

Not really, in the same way that you don't list pip in your dependencies. :-) uv is the installer so it needs to exist outside of the project.

agoscinski

Thanks again for the work @danielhollas

unkcpz · 2024-12-05T17:16:33Z

I remove the outdated "check-requirement" CI restriction. So it is able to merge.
This is a great CI improvement, thanks a lot for the effort! @danielhollas

unkcpz · 2024-12-05T17:22:45Z

It seems other CI actions are failed after merging this. I'll have a look later.

unkcpz · 2024-12-05T17:59:30Z

After this PR, the AEP https://github.com/aiidateam/AEP/blob/master/002_dependency_management/readme.md is outdated. @danielhollas would you mind to update it to sync it (or put a link in the AEP point to the doc) with the developer documentation you are preparing?

GeigerJ2 · 2024-12-06T10:18:55Z

Sorry, @danielhollas, for ignoring this... been busy with some other stuff. I see the rest of the aiida-core office took over. Thanks for the work! <3

danielhollas · 2024-12-06T12:54:30Z

No worries at all, I was mostly tagging everybody for visibility. :-)

unkcpz · 2024-12-20T15:51:52Z

@danielhollas I have to say, this is a fantastic change. It makes testing against different branch extremely easy. I benefit a lot when working and test with plumpy/kiwipy. +100 kudos and thanks again.

danielhollas force-pushed the uv-lock branch from 5b73199 to a3ef407 Compare November 26, 2024 14:09

WIP: See if we can use uv lockfile

dff7bde

danielhollas force-pushed the uv-lock branch from a3ef407 to dff7bde Compare November 26, 2024 14:27

danielhollas added 3 commits November 26, 2024 14:34

Add pre-commit extra

368dff3

Activate the environment

63f9929

s/from-requirements/from-lock/

924f1bb

danielhollas mentioned this pull request Nov 26, 2024

Transport & Engine: AsyncTransport plugin #6626

Open

danielhollas changed the title ~~WIP: See if we can use uv lockfile~~ See if we can use uv lockfile Nov 28, 2024

danielhollas added 8 commits November 28, 2024 18:51

git rm -r requirements/

47095d6

Merge branch 'main' into uv-lock

0e3b87b

Validate lockfile in CI

954b0ce

Add pre-commit hook for checking uv lockfile

166c978

In this implementation, the hook will only check whether the lockfile is compatible with pyproject.toml, it will not try to automatically update it, since that should be decided by the developer.

Fix pre-commit config

b0cadaf

Remove check-requirements job

3517d65

Update CODEOWNERS file

e3fb04d

CI: Remove needless jobs

10cd62a

danielhollas commented Nov 28, 2024

View reviewed changes

.github/CODEOWNERS Show resolved Hide resolved

danielhollas commented Nov 28, 2024

View reviewed changes

danielhollas marked this pull request as ready for review November 28, 2024 20:42

danielhollas changed the title ~~See if we can use uv lockfile~~ CI: Utilize uv lockfile for reproducible test environments Nov 28, 2024

danielhollas requested review from unkcpz and agoscinski November 28, 2024 21:04

unkcpz reviewed Dec 1, 2024

View reviewed changes

Update lock

c5c63a2

Update CODEOWNERS

352539d

danielhollas force-pushed the uv-lock branch from 5cc8846 to 352539d Compare December 2, 2024 12:21

danielhollas requested review from khsrali and GeigerJ2 December 2, 2024 12:34

agoscinski reviewed Dec 3, 2024

View reviewed changes

.github/workflows/test-install.yml Show resolved Hide resolved

agoscinski requested changes Dec 3, 2024

View reviewed changes

danielhollas added 3 commits December 4, 2024 11:30

Merge branch 'main' into uv-lock

16805af

Add comments

7959c4d

Bump uv version

e62a2ef

agoscinski approved these changes Dec 4, 2024

View reviewed changes

unkcpz mentioned this pull request Dec 5, 2024

Incompatible version of typing-extensions with those in aiida-core aiidateam/plumpy#294

Open

unkcpz merged commit 04cc344 into aiidateam:main Dec 5, 2024
22 checks passed

danielhollas deleted the uv-lock branch December 5, 2024 17:21

unkcpz mentioned this pull request Dec 5, 2024

CI: Test with RabbitMQ 4.0.x #6649

Merged

This was referenced Dec 5, 2024

CI: quick fix on benchmark CI using uv run pytest #6651

Closed

CI: quick fix on benchmark CI using uv run pytest #6652

Merged

		utils/dependency_management.py @aiidateam/dependency-manager
		.github/workflows/dm.yml @aiidateam/dependency-manager

	run: source .venv/bin/activate && .github/workflows/setup.sh
	run: uv venv && source .venv/bin/activate && .github/workflows/setup.sh

CI: Utilize uv lockfile for reproducible test environments #6640

CI: Utilize uv lockfile for reproducible test environments #6640

Conversation

danielhollas commented Nov 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielhollas commented Nov 28, 2024

codecov bot commented Nov 28, 2024 • edited Loading

Codecov Report

unkcpz commented Nov 29, 2024 • edited Loading

khsrali commented Nov 29, 2024 • edited Loading

danielhollas commented Nov 29, 2024

unkcpz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielhollas commented Dec 2, 2024

danielhollas commented Dec 2, 2024

danielhollas commented Dec 2, 2024

unkcpz commented Dec 2, 2024

agoscinski left a comment • edited Loading

Choose a reason for hiding this comment

agoscinski commented Dec 3, 2024

agoscinski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danielhollas commented Dec 4, 2024

khsrali commented Dec 4, 2024

danielhollas commented Dec 4, 2024

agoscinski left a comment

Choose a reason for hiding this comment

unkcpz commented Dec 5, 2024

unkcpz commented Dec 5, 2024

unkcpz commented Dec 5, 2024

GeigerJ2 commented Dec 6, 2024

danielhollas commented Dec 6, 2024

unkcpz commented Dec 20, 2024

danielhollas commented Nov 25, 2024 •

edited

Loading

codecov bot commented Nov 28, 2024 •

edited

Loading

unkcpz commented Nov 29, 2024 •

edited

Loading

khsrali commented Nov 29, 2024 •

edited

Loading

agoscinski left a comment •

edited

Loading