feat: make lockfile contain urls instead of filenames

[betaboon]

Pull Request Check List

• A news fragment is added in news/ describing what is new.
• Test cases added for changed code.

Describe what you have changed in this PR.

This PR changes the lockfile to contain full urls instead of just the filenames.

Context

I'm coming from efforts in the bazel world to properly integrate cross-platform python-lockfiles. (the effort in question is here )
There we stumbled upon the problem that it is hard to resolve filenames in lockfiles to urls and it would be alot easier if the lockfiles would already include the urls, as the tool creating the lockfile should already have the knowledge about the urls.

I hacked together a little patch the does that with pdm and talked to @frostming on discord, who signaled interest in this change.

so here we are ;)

[pawamoy]

Hello @betaboon, is there an open issue that discusses this change? If so, could you link to it in the PR description? I'd like to understand the reasoning behind it 🙂 I think NPM puts the resolved URL in the lock file as well, maybe you got inspired by other package managers?

[betaboon]

Hello @betaboon, is there an open issue that discusses this change? If so, could you link to it in the PR description? I'd like to understand the reasoning behind it slightly_smiling_face I think NPM puts the resolved URL in the lock file as well, maybe you got inspired by other package managers?

@pawamoy i added some context-explanation to the description. hope that clears it up a little :)

[frostming]

@pawamoy The discussion actually happens in discord and it SGTM as the filename is not being used after all by now.

@betaboon Can you bump the minor version at

pdm/pdm/project/core.py

Line 65 in cc2ad1e

LOCKFILE_VERSION = "3.1"

?

[betaboon]

@pawamoy The discussion actually happens in discord and it SGTM as the filename is not being used after all by now.

@betaboon Can you bump the minor version at

pdm/pdm/project/core.py

Line 65 in cc2ad1e

LOCKFILE_VERSION = "3.1"

?

done

[frostming]

@pawamoy The discussion actually happens in discord and it SGTM as the filename is not being used after all by now.
@betaboon Can you bump the minor version at

pdm/pdm/project/core.py

Line 65 in cc2ad1e

LOCKFILE_VERSION = "3.1"

?

done

I am sorry, it seems we should bump the major version, let's make it "4", since it can't be read by old PDM versions. Or if you keep the filename it can be "3.2"

[betaboon]

Which one do you prefer? I can do either.

I think minor (aka adding URL instead of replacing filename) might prevent some migration problems for ppl.

On the other hand since you're aiming for a 2.0 release might be the perfect time to introduce a "breaking" change

[frostming]

Which one do you prefer? I can do either.

At this point it is okay to make a major bump, the PDM can regenerate the lock file automatically if it sees an incompatible lock file version when doing installation.

[pawamoy]

@betaboon Thanks!
@frostming oh I see, thanks, I rarely check Discord 🙂

[betaboon]

bumped LOCKFILE_VERSION to 4.0

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.20%. Comparing base (204f96c) to head (07861ca).
Report is 1032 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1203      +/-   ##
==========================================
+ Coverage   81.08%   81.20%   +0.12%     
==========================================
  Files          93       93              
  Lines        7596     7597       +1     
  Branches     1792     1792              
==========================================
+ Hits         6159     6169      +10     
+ Misses       1082     1076       -6     
+ Partials      355      352       -3     

Flag	Coverage Δ
unittests	81.08% <100.00%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[frostming]

The hash seems duplicating, maybe some bug?

[betaboon]

i noticed that yesterday to.
this is actually due to: https://github.com/pdm-project/pdm/blob/main/pdm/project/config.py#L36-L39

since there are two default-repositories, there are two hashes for each file (one for each repository).

the way to prevent duplicate hashes would be using a set here. but that would change the order of hashes.

[frostming]

this is actually due to: https://github.com/pdm-project/pdm/blob/main/pdm/project/config.py#L36-L39

I doubt that, though a bit confusing, the default repositories are for uploading packages, and have nothing to do with downloading and fetching

[betaboon]

if you take a look at the generated pdm.lock it also contains files from both repositories

[frostming]

I see, so it is because of the source settings:

[[tool.pdm.source]]
url = "https://test.pypi.org/simple"
verify_ssl = true
name = "testpypi"

[betaboon]

ah yeah you're right. it's due to pyproject.toml in the example projects.

i think we should only do deduplication of hashes in the requirements.txt
i see value in having multiple urls for the same package in the pdm.lock as it allows fallback. imagine the case where the first source is available at lock-time, not not available at install time.

[frostming]

imagine the case where the first source is available at lock-time, not not available at install time.

Sounds sensible, then we will implement the behavior I mentioned in the gist:

When installing from the lock file, take the links stored with the lock file.

[betaboon]

two questions:

• should i add hash-deduplication in requirements.txt-export as mentioned above?
• can you take care of the "take links from lock file during install" (in a seperate PR)?

[frostming]

First, of course
Second, of course

[betaboon]

done

[frostming]

LGTM, can you please add a release note under news folder? I've added one.