[feature] ability to update a reference to an earlier version in a conan lockfile

[jcar87]

What is your suggestion?

From: https://cpplang.slack.com/archives/C41CWV9HA/p1692362444235349

At the moment, it is possible to update a lockfile to a new version (and also update the revision), but with conan lock add it's not possible to add an earlier version in a meaningful way (it would not resolve to that one).

Some options:

• conan lock remove + conan lock add + conan lock create
• conan lock update (some challenges: lockfile may have different reversions/revisions of the same recipe, which one gets updated, consider a pattern)

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[antoinebardoz]

A related issue is that without --require-override there is now no way to get packages that don't conform to the version range.

In our specific case, the issue is that we build test packages with a "test" user, and we want to trigger CI-pipelines for other upstream packages. We used to be able to do this in a very simple way:

conan create upstream_package --require-override dependency/version@test

Even if the version range in upstream_package/conanfile.py was:

self.requires("dependency/[>=1.0 < 2.0]@company_name")

Now, even if I override the lock file to include the specific version of the test package, I get:

ERROR: Requirement 'dependency/[>=1.0 < 2.0]@company_name' not in lockfile

because the user is @test.

So we went from a situation where overriding versions was convenient and simple to nearly impossible. Do I have to go back to using sed to overwrite the conanfile?

EDIT: While I think it (obviously) makes sense to adhere to version ranges in the general case, it's incredibly convenient and useful to be given the option to say "use this package, no matter what the version ranges are".

[memsharded]

Hi @antoinebardoz

There is underlying thing there, which is the practice of using channels as a variable controlling the flow and usage of packages.
This is no longer recommended, we have been discouraging this usage for some time already, and encouraging instead the usage of different server repositories to control which packages are resolved.

Conan 2.0 also proposes other mechanisms to achieve this, for example, in order to test "test", the right approach would be to use semver pre-releases, and activate it with the core.version_ranges:resolve_prereleases conf, for automatically resolving to those, without needing to change a single line anywhere.

This is way more conforming to standard versioning practices and it is the recommended way moving forward.

[antoinebardoz]

Hi @memsharded,

Thank you for your response.

Indeed, we are now experimenting with using a different method than "user" to differentiate between test packages and release packages (I did not use channels). We will now try using semantic versioning instead.

However, I think my point still stands. It is extremely useful to be able to easily force behavior when necessary. The default behavior should be to conform to the version ranges, but if a user wants to test something (perhaps even test it locally on their machine), it should also be easy for them to force their entire build graph to use a certain version (with --build=missing, if necessary). This would greatly reduce the mental load of doing normal things, such as:

• Checking that every package in your build graph can build with your latest changes (especially if your package is a dependency of several other packages in your graph)
• Being able to set very "special" version numbers in your test packages, so they are never accidentally included in a release build because they purposefully do not conform to the version ranges.

I agree correct versioning use should be encouraged. However, hacky use should be allowed when requested, and perhaps a warning could be emitted rather than a failure. I cannot stress enough how important it is to reduce the mental overhead during development. Conan is not an easy tool to use, even for me who's been actively using it for years. Now imagine how it is for my team members who are novices.

Now that I have written my own script to do require-override in the lockfile, and have set appropriate version ranges, it works as intended (by the way, I was very impressed that now I can override a dependency of a dependency without having to rebuild the direct dependency. This is a major step forward for Conan 2). But it would've been really awesome if I didn't have to write this script myself.

[memsharded]

However, I think my point still stands. It is extremely useful to be able to easily force behavior when necessary. The default behavior should be to conform to the version ranges, but if a user wants to test something (perhaps even test it locally on their machine), it should also be easy for them to force their entire build graph to use a certain version (with --build=missing, if necessary). This would greatly reduce the mental load of doing normal things, such as:

But this is easy and straightforward. If a developer wants to test a certain thing in their machine, the only thing they need to do is to build/create/export what they want to their local cache, with the same name and version (or compatible version in the range). Conan graph resolution will be using their own cache package automatically. No need to override, not need to use lockfiles, or change anything anywhere. They can also use editables to not need to export it to the cache, and use their own local copy in their user space, that they can easily modify code, debug, etc.

I agree correct versioning use should be encouraged. However, hacky use should be allowed when requested, and perhaps a warning could be emitted rather than a failure. I cannot stress enough how important it is to reduce the mental overhead during development. Conan is not an easy tool to use, even for me who's been actively using it for years. Now imagine how it is for my team members who are novices.

We agree with this, and it has been one of the main reasons to move in this direction in Conan 2.0. The use of variable user/channel part is one of the things that brought a lot of developer mental overhead, confusion and problems to tons of developers. We know it because we not only respond to the thousands of tickets along the years here in Github, but we also do lots of videocalls and other engagements with hundreds of teams and companies using Conan. So we decided long time ago to stop using user/channel as extra "version" qualifiers to remove this overhead, and centralized the behavior in the version and version-ranges concepts that developers are already familiar with.

Now that I have written my own script to do require-override in the lockfile, and have set appropriate version ranges, it works as intended (by the way, I was very impressed that now I can override a dependency of a dependency without having to rebuild the direct dependency. This is a major step forward for Conan 2).

Thanks, I am very happy that you appreciate some of the improvements that Conan 2.0 brings. We strongly believe that 2.0 is a much better, order of magnitude, Conan version, with tons of improvements over the graph, lockfiles, UX, command line, extensibility, etc. And we also know that it can be frustrating when something that used to work in the past is removed, and we have considered those removals quite thoroughly, and in this case we were sure that the conclusion of dropping the flows that use user/channel as variable coordinates to reference (and override) packages is another improvement that will help the wider community to simplify things and converge to more standard practices in package management, while reducing the overhead of future Conan users.

But it would've been really awesome if I didn't have to write this script myself.

Sure, so far this feature is looking good, and we will add commands to do this task automatically.

Thanks for all the feedback!

[antoinebardoz]

But this is easy and straightforward. If a developer wants to test a certain thing in their machine, the only thing they need to do is to build/create/export what they want to their local cache, with the same name and version (or compatible version in the range). Conan graph resolution will be using their own cache package automatically. No need to override, not need to use lockfiles, or change anything anywhere. They can also use editables to not need to export it to the cache, and use their own local copy in their user space, that they can easily modify code, debug, etc.

Hmmm, I will try to look into this more closely, we haven't been very good at using editable so far, even though that is my most favoured option (it requires some setup, and I suspect we will have to do some restructuring of some packages). With regards to using the local cache, I will try to understand better how packages are chosen, and my impression is that this could be easier in Conan 2 than 1. In 1, my impression was that it wasn't very simple.

One thing I found painful when using editable in the past (Conan 1) is that the version range of the editable package was still taken into account. If a package exists in some repo which is of a higher version it may be chosen instead of the editable package. In my mind, if I say "use this package as editable" Conan should just do it, and not worry about the version of the package (or at the very least there should be a --force option). I haven't tried this in Conan 2 yet, so perhaps this has changed, but if not I think this is something to consider.

Speaking of user/channel, we have now moved on from that, and have started using semver to distinguish between release and pre-release. In principle, I agree it is better than user/channel, none of us were particularly fond of that anyway.

However, I ran into a little snag, which may be relevant to this ticket. With our current flow, we use a script to overwrite the version of a dependency in the lockfile, to test pre-releases. For example:
dependency/1.0.0 could be overridden to dependency/1.0.0-pre.

The issue I ran into was that our range was:
[>=1.0.0 <2], and Conan refused to resolve this, presumably because 1.0.0-pre is a smaller version than 1.0.0. I understand the logic of this, but in my mind, if I have resolve_prereleases set to true, Conan should include pre-releases at the lower end of the range.

Perhaps the feature from this ticket would resolve this case anyway?

[memsharded]

Hmmm, I will try to look into this more closely, we haven't been very good at using editable so far, even though that is my most favoured option (it requires some setup, and I suspect we will have to do some restructuring of some packages)

With the correct layout() this should be out of the box. For example the templates like conan new cmake_lib -d name=mypkg -d version=0.1 are editable out of the box. Also other 2.0 utilities like --build=editable allows to do an ordered build of all editables in the right order, in user space, which can be very convenient. In Conan 1, with the layout.ini files it was not simple at all, but quite complicated and fragile.

One thing I found painful when using editable in the past (Conan 1) is that the version range of the editable package was still taken into account. If a package exists in some repo which is of a higher version it may be chosen instead of the editable package. In my mind, if I say "use this package as editable" Conan should just do it, and not worry about the version of the package (or at the very least there should be a --force option). I haven't tried this in Conan 2 yet, so perhaps this has changed, but if not I think this is something to consider.

Conan 2.0.10 has gotten resolution of version ranges taking into account the editables in #14510. It is not absolute, in the sense that if there is package in the cache with a more modern version than the editable it will be picked. This was absolutely necessary, because a global override cannot happen, there are some scenarios in which different versions of the same package are used in the same graph, so it should be possible to have both mypkg/1.0 and mypkg/2.2 in editable simultaneously, and let different consumers of those resolve to the respective one based on their version ranges.

In practice this should be a big inconvenience, if you want to force it, you could either remove the newer versions from the Conan cache, or bump the editable recipe version to something newer, because you are editing the recipe locally, this should be quite straightforward.

In any case adding #14510 was a missing piece, very necessary for resolving version ranges for editables, please update to 2.0.10, feedback welcome.

The issue I ran into was that our range was:
[>=1.0.0 <2], and Conan refused to resolve this, presumably because 1.0.0-pre is a smaller version than 1.0.0. I understand the logic of this, but in my mind, if I have resolve_prereleases set to true, Conan should include pre-releases at the lower end of the range.

I am afraid this sound quite breaking, and will most likely cause lots of confusion and issues to other Conan users. The pre-releases approach in semver does not really alter the definition of the version range. 2.0.0-pre.1 is lower than <2, so it totally belongs to the defined range [>=1.0.0 <2]. But pre-release is a qualifier that means exactly that for semver: "even if it is in the desired range, do not use it unless I explicitly say so". Trying to resolve to 1.0.0-pre.1 that is outside of the defined range will be incredibly breaking, scenario:

• Users have requires = "mydep/[>=1.0.0 <2]" in their dependencies (and many other similar dependencies)
• Typical builds are already resolving to mydep/1.7.2, as the requires have been defined for a while without problems
• Users have the nightly flow of activating core.version_ranges:resolve_prereleases=True to check if some team releasing some mydep/2.0.0-alfa.1 (or any other dependency) and test it.
• Now, for all the teams not releasing a higher bound pre-release, the system will bring all the 1.0.0-pre.1 pre-releases. They are very unlikely to play well together with all the other more modern dependencies, at the end of the day, they were pre-releases used to polish the final accepted 1.0.0, it is very likely they have breaking bugs that were detected back then, and are no longer acceptable.

[antoinebardoz]

Hi @memsharded,

I have now been thinking about this for several weeks, and have been taking your feedback into account. In some cases I agree with you, and we've been able to work through several of the issues we've had.

However, I seem to keep running into new issues when it comes to versioning. I'm sorry that this post is very long, but it's quite difficult to explain succinctly.

With regards to issues we have already discussed:

1. Overriding user/channel: We can instead use pre-releases as you suggested. I consider this matter resolved.
2. Forcing use of editable: While I still think it would be nice to be given the option to --force, as it is for local use it can be worked around by setting an appropriate version. I consider this matter mostly resolved.
3. Pre-release ranges: We can find a solution to this by setting the version range to one less than the pre-release. I consider this to be resolved.
4. Being able to override versions in lockfiles: Seemingly, to me, this is still a major problem.

Considering issue 4:
First of all, I view both the present issue (14523) and #14524 to both be related to this. And most of all, this is relevant to us in CI.

We have a workflow where if package TOP depends on package DEP, the CI pipeline of package DEP will trigger a CI pipeline for TOP. This pipeline will override the lockfile of TOP to use the version of DEP under test. We wrote a little conan extension to do this override, which simply iterates through the TOP/conan.lock (we are now checking lockfiles into git) and replaces the line containing DEP/version_used_for_TOP with DEP/version_under_test.

This works okay, and has been running for a few weeks. However, we have some concerns. Several concerns.

Does this CI flow make sense?

We have been using this pipeline triggering flow for years, and it has been very laborious, and has caused many difficulties with Conan. Are we doing something wrong? Sometimes I get the feeling that nobody else is doing this since we have so many corner cases for which there are no pre-existing solutions. On the other hand, it's hard to understand how else we should be doing it.

Transient dependencies

If DEP has another dependency DEPDEP, and we try to override TOP/conan.lock, we end up only overriding the version of DEP and not the version of DEPDEP. Fixing this would require a complex solution which we believe should be beyond the scope of our little utility extension. However, being given no choice at the moment, we are in the process of writing an extension which does this.

We suggest:
conan lock add --require-override=DEP/version_under_test

Which would not add a second entry of DEP, but overwrite the entry which is in the existing lockfile, and also overwrite the version of DEPDEP with the version used to build DEP.

Merging lockfiles with priorities

Currently, there doesn't seem to be any way at all to override the lockfile for TOP with the versions used to build DEP. We end up with multiple versions of the same packages (if we use lock merge or add). E.g:

conan lock merge TOP/conan.lock DEP/conan.lock > result.lock

Leads to something like:

result.lock:

requires = [
       DEP/version_under_test
       DEPDEP/version_used_for_TOP
       DEPDEP/version_used_for_DEP
]

In this case, I am really mostly interested in testing DEP. I'm testing that it still works with TOP, so that I will know ahead of time if there are breaking changes. The problem I have with the way things are now, is that I depend on DEPDEP/version_used_for_DEP having a higher version than DEPDEP/version_used_for_TOP, which is very complicated to automate, especially since we may be dealing with several pre-releases.

So what I want, really, is:

result.lock:

requires = [
       DEP/version_under_test
       DEPDEP/version_used_for_DEP
]

We suggest:
conan lock merge --lockfile_hi_pri=DEP/conan.lock --lockfile_lo_pri=TOP/conan.lock --lockfile_out=result.lock

Which would, if there are dependencies which are listed both DEP/conan.lock and TOP/conan.lock, choose the versions listed in DEP/conan.lock.

[memsharded]

This has been implemented in #15284, with the following approach, using a conan lock remove new subcommand, that allows to explicitly define updates.

The idea is that updates are equivalent to something like:

$ conan remove --requires=zlib/* && conan add --requires=zlib/1.2.11

Please see description of #15284 for more info.