Feature/versioning WIP

conf.py

@@ -397,6 +397,8 @@ def copy_legacy_redirects(app, docname): # Sphinx expects two arguments
         "creating_packages/package_dev_flow.html": "../developing_packages/package_dev_flow.html",
         "conan1.0.html": "faq/conan1.0.html",
         "mastering/python_requires.html": "../extending/python_requires.html",
+        "mastering/version_ranges.html": "../versioning/version_ranges.html",
+        "mastering/revisions.html": "../versioning/revisions.html",
 
         "integrations/cmake.html": "build_system/cmake.html",
         "integrations/makefile.html": "build_system/makefile.html",

index.rst

@@ -19,6 +19,7 @@ Contents:
    uploading_packages
    developing_packages
    devtools
+   versioning
    mastering
    systems_cross_building
    extending

mastering.rst

@@ -10,11 +10,9 @@ This section provides an introduction to important productivity and useful featu
 
    mastering/conanfile_py
    mastering/conditional
-   mastering/version_ranges
    mastering/policies
    mastering/envvars
    mastering/virtualenv
    mastering/logging
    mastering/sharing_settings_and_config
    mastering/custom_cache
-   mastering/revisions

reference/conanfile/methods.rst

@@ -442,6 +442,8 @@ it is an example of a recipe for a library that doesn't support Windows operatin
 
 This exception will be propagated and Conan application will finish with a :ref:`special return code <invalid_configuration_return_code>`.
 
+.. _method_requirements:
+
 requirements()
 --------------
 

versioning.rst

@@ -0,0 +1,12 @@
+.. _versioning:
+
+Versioning
+----------
+
+.. toctree::
+   :maxdepth: 2
+
+   versioning/introduction
+   versioning/version_ranges
+   versioning/revisions
+   versioning/lockfiles

versioning/introduction.rst

@@ -0,0 +1,153 @@
+.. _versioning_introduction:
+
+Introduction to versioning
+==========================
+
+Versioning approaches
+---------------------
+
+Fixed versions
+++++++++++++++
+
+This is the standard, direct way to specify dependencies versions, with their exact
+version, for example in a *conanfile.py* recipe:
+
+.. code-block:: python
+
+    requires = "zlib/1.2.11@conan/stable"
+
+When doing a :command:`conan install`, it will try to fetch from the remotes exactly
+that *1.2.11* version.
+
+This method is nicely explicit and deterministic, and is probably the most used one.
+As a possible disadvantage, it requires the consumers to explicitly modify the recipes
+to use updated versions, which could be tedious or difficult to scale for large projects
+with many dependencies, in which those dependencies are frequently modified, and
+it is desired to move the whole project forward to those updated dependencies.
+
+Version ranges
+++++++++++++++
+
+A *conanfile* can specify a range of valid versions that could be consumed, using brackets:
+
+..  code-block:: python
+
+      requires = "pkg/[>1.0 <1.8]@user/stable"
+
+When a :command:`conan install` is executed, it will check in the local cache first and if
+not in the remotes what ``pkg`` versions are available and will select the latest one
+that satisfies the defined range.
+
+By default, it is less deterministic, one ``conan install`` can resolve to ``pkg/1.1`` and
+then ``pkg/1.2`` is published, and a new ``conan install`` (by users, or CI), will automatically
+pick the newer 1.2 version, with different results. On the other hand it doesn't require
+changes to consumer recipes to upgrade to use new versions of dependencies.
+
+It is also true that the *semver* definition that comes from other programming languages
+doesn't fit that well to C and C++ packages, because of different reasons, because of 
+open source libraries that don't closely follow the semver specification, but also because
+of the ABI compatibility issues and compilation model that is so characteristic of C and C++ binaries.
+
+Read more about it in :ref:`version_ranges` section.
+
+Package alias
++++++++++++++
+
+It is possible to define a "proxy" package that references another one, using the syntax:
+
+.. code-block:: python
+
+    from conans import ConanFile
+
+    class AliasConanfile(ConanFile):
+        alias = "pkg/0.1@user/testing"
+
+This package creation can be automatically created with the :ref:`conan_alias` command, that
+can for example create a ``pkg/latest@user/testing`` alias that will be pointing to that
+``pkg/0.1@user/testing``. Consumers can define ``requires = "pkg/latest@user/testing"`` and
+when the graph is evaluated, it will be directly replaced by the ``pkg/0.1`` one. That is,
+the ``pkg/latest`` package will not appear in the dependency graph at all.
+
+This is also less deterministic, and puts the control on the package creator side, instead of
+the consumer (version ranges are controlled by the consumer). Package creators can control
+which real versions will their consumers be using. This is probably not the recommended way
+for normal dependencies versions management.
+
+
+Package revisions
++++++++++++++++++
+
+Revisions are automatic internal versions to both recipes and binary packages.
+When revisions are enabled, when a recipe changes and it is used to 
+create a package, a new recipe revision is generated, with the hash of the
+contents of the recipe. The revisioned reference of the recipe is:
+
+.. code-block:: text
+
+    pkg/version@user/channel#recipe_revision1
+    # after the change of the recipe
+    pkg/version@user/channel#recipe_revision2
+
+A conanfile can reference a specific revision of its dependencies, but in
+the general case that they are not specified, it will fetch the latest
+revision available in the remote server:
+
+.. code-block:: text
+
+    [requires]
+    # Use the latest revision of pkg1
+    pkg1/version@user/channel 
+    # use the specific revision RREV1 of pkg2
+    pkg2/version@user/channel#RREV1
+
+Each binary package will also be revisioned. The good practice is to build each
+binary just once. But if for some reason, like a change in the environment, a new
+build of exactly the same recipe with the same code (and the same recipe revision)
+is fired again, a new package revision can be created. The package revision
+is the hash of the contents of the package (headers, libraries...), so unless
+deterministic builds are achieved, new package revisions will be generated.
+
+In general revisions are not intended to be defined explictly in conanfiles,
+altough they can for specific purposes like debugging.
+
+Read more about :ref:`package_revisions`
+
+
+Version and configuration conflicts 
+-----------------------------------
+
+When two different branches of the same dependency graph require the same package,
+this is known as "diamonds" in the graph. If the two branches of a diamond require
+the same package but different versions, this is known as a conflict (a version conflict).
+
+Lets say that we are building an executable in **PkgD/1.0**, that depends on **PkgB/1.0** and **PkgC/1.0**,
+which contain static libraries. In turn, **PkgB/1.0** depends on **PkgA/1.0** and finally **PkgC/1.0** depends on 
+**PkgA/2.0**, which is also another static library.
+
+The executable in **PkgD/1.0**, cannot link with 2 different versions of the same static library in **PkgC**, and the dependency resolution algorithm raises an error to let the
+user decide which one.
+
+.. image:: ../images/graph_conflicts.png
+
+The same situation happens if the different packages require different configurations of the same upstream package, even if the same version is used. In the example above, both **PkgB** and **PkgC** can be requiring the same version **PkgA/1.0**, but one of them will try to use it as a static library and the other one will try to use it as shared library. 
+The dependency resolution algorithm will also raise an error.
+
+Dependencies overriding
+-----------------------
+The downstream consumer packages always have higher priority, so the versions they request, will be overriden upstream as the dependency graph is built, re-defining the possible requires that the packages could have. For example, **PkgB/1.0** could define in its recipe a dependency to **PkgA/1.0**. But if a downstream consumer defines a requirement to **PkgA/2.0**, then that version will be used in the upstream graph:
+
+.. image:: ../images/graph_override.png
+
+This is what enables the users to have control. Even when a package recipe upstream defines an older version, the downstream consumers can force to use an updated version. Note that this is not a diamond structure in the graph, so it is not a conflict by default. This behavior can be also restricted defining the :ref:`env_vars_conan_error_on_override` environment variable to raise an error when these overrides happen, and then the user can go and explicitly modify the upstream **PkgB/1.0** recipe to match the version of PkgA and avoid the override.
+
+In some scenarios, the downstream consumer **PkgD/1.0** might not want to force a dependency on PkgA. There are several possibilities, for example that PkgA is a conditional requirement that only happens in some operating systems. If PkgD defines a normal requirement to PkgA, then, it will be introducing that edge in the graph, forcing PkgA to be used always, in all operating systems. For this purpose the ``override`` qualifier can be defined in requirement, see :ref:`method_requirements`.
+
+
+Versioning and binary compatibility
+-----------------------------------
+
+It is important to note and this point that versioning approaches and strategies should also be
+consistent with the binary management. 
+
+By default conan assumes *semver* compatibility, so it will not require to build a new binary for a package when its dependencies change their minor or patch versions. This might not be enough for C or C++ libraries which versioning scheme doesn't strictly follow semver. It is strongly suggested to read more about this in :ref:`define_abi_compatibility`
+

versioning/lockfiles.rst

@@ -0,0 +1,176 @@
+.. _versioning_lockfiles:
+
+Lockfiles
+=========
+
+.. warning::
+
+    This is an **experimental** feature subject to breaking changes in future releases.
+
+
+Lockfiles are files that store the information of a dependency graph, including the
+exact versions, revisions, options, and configuration of that dependency graph. As 
+they depend on the configuration, and the dependency graph can change with every 
+different configuration, there will be one lockfile for every configuration.
+
+Lockfiles are useful for achieving deterministic builds, even if the dependency 
+definitions in conanfile recipes are not fully deterministic, for example when using
+version ranges or using package revisions.
+
+Let's say we have 3 package recipes PkgC, PkgB, and PkgA, that define this dependency graph:
+
+.. image:: ../images/graph_not_deterministic.png
+
+The first time, when a :command:`conan install .` is executed, the requirement defined
+in PkgB is resolved to **PkgA/1.0**, because that was the latest at that time that
+satisfied the version range ``PkgA/[*]``. After such install, the user can build and 
+run an application in the source code of PkgC. But some time later, another colleague
+tries to do exactly the same, and suddenly it is pulling a newer version of PkgA that
+was recently published, getting different results (maybe even not working). Builds with
+version ranges are not reproducible by default.
+
+Using lockfiles
+---------------
+Lockfiles solve this problem creating a file that stores this information. In the example
+above, the first :command:`conan install .` will generate a *conan.lock* file that can be
+used later:
+
+.. code-block:: bash
+
+    $ cd PkgC
+    $ conan install .  # generates conan.lock
+    # After PkgA/1.1 has been created
+    $ conan install . --lockfile # uses the existing conan.lock
+
+The second time that :command:`conan install . --lockfile` is called, with the lockfile argument
+it will load the previously generated *conan.lock* file, that contains the information that
+**PkgA/1.0** is used, and will apply it again to the dependency resolution, resolving exactly
+the same dependency graph:
+
+.. image:: ../images/graph_locked.png
+
+The *conan.lock* file contains more information than the versions of the dependencies, it contains:
+
+- The "effective" profile that has been used to compute this lockfile. The effective profile is the
+  combination of the profile files that could have been passed in the command line, pluse any
+  other settings or options directly defined in the command line.
+- It encodes a graph, not just a list of versions, as different nodes in the graph might be using
+  different versions too.
+- The options values at each package. As downstream consumers can define options values, it is
+  important that this information is also stored, so it is also possible to build intermediate nodes
+  of the graph leading to the same result.
+- Other kind of requirements like python_requires.
+
+Again, it is important to remember that every different configuration will generate a different
+graph, and then a different *conan.lock* as result. So the example above would be more like the
+following if we wanted to work with different configurations (e.g. Debug/Release):
+
+.. code-block:: bash
+
+    $ cd PkgC
+    $ cd release
+    $ conan install .. # generates conan.lock (release) in this folder
+    $ cd ../debug
+    $ conan install .. -s build_type=Debug # generates conan.lock (debug) 
+    # After PkgA/1.1 has been created
+    $ conan install .. --lockfile # uses the existing conan.lock (debug)
+    $ cd ../release
+    $ conan install .. --lockfile # uses the existing conan.lock (release)
+
+
+Commands
+--------
+
+There are 2 main entry points for lockfile information in conan commands:
+
+- :command:`--lockfile` argument in :command:`install/create/export/info`
+
+    If the command builds a package, it can modify its reference. Even if teh version is not changed,
+    if something in the recipe changes, it will get a new recipe revision RREV and if the package is
+    built from sources again, it might end with a new, different package revision PREV. Those changes
+    will be updated in the *conan.lock* lockfile, and the package will be marked as "modified".
+
+- :command:`conan graph` command
+
+    This command group contains several functions related to the management of lockfiles:
+
+    - :command:`conan graph lock`
+
+        This command will generate a *conan.lock* file. It behaves like :command:`conan install` command,
+        (this will also generate a lockfile by default), but without needing to actually install the
+        binaries, so it will be faster. In that regard it is equal to :command:`conan info` that can also
+        generate a lockfile, but the problem with :command:`conan info -if=.` is that it does not allow to 
+        specify a profile or settings.
+
+    - :command:`conan graph clean-modified`
+
+        When a :command:`conan create` command that uses a lockfile builds a new binary, its reference
+        will change. This change, typically in the form of a recipe revision and/or package revision
+        is updated in the lockfile and the node is marked as "modified". This :command:`clean-modified`
+        removes these "modified" flags from a lockfile. This operation is typically needed before starting
+        the build of a package in a locked graph, to know exactly which nodes have been modified by this
+        operation.
+
+    - :command:`conan graph update-lock`
+
+        Update the current lockfile with the information of the second lockfile. Only the nodes marked
+        as "modified" will be updated. Trying to update to the current lockfile one node that has already
+        been "modified" will result in an error.
+
+    - :command:`conan graph build-order`
+
+        Takes a lockfile as argument, and return a list of lists indicating the order in which packages
+        in the graph have to be built. It only returns those packages that really need to be built,
+        following the :command:`--build` arguments and the ``package_id()`` rules.
+
+
+For more information see :ref:`commands`
+
+
+How to use lockfiles in CI
+--------------------------
+
+One of the applications of lockfiles is to be able to propagate changes done in one package
+belonging to a dependency graph downstream its affected consumers.
+
+Lets say that we have the following scenario in which one developer does some changes to PkgB,
+creating a new version **PkgB/1.1**, and lets assume that all packages use version-ranges.
+The goal is to be able to build a new dependency graph down to **PkgE/1.0** in which only those
+changes done in **PkgB/1.1** are taken into account but not other new versions of other packages,
+like new versions of PkgA and PkgF.
+
+The process can begin capturing a lockfile of the initial status of our "product", down to PkgE:
+
+.. code-block:: bash
+
+    $ cd PkgE
+    $ conan graph lock .
+
+After that, we can proceed to build the new **PkgB/1.1** version, with its dependencies locked:
+
+.. code-block:: bash
+
+    $ cd PkgB
+    $ cp ../PkgE/conan.lock . # Do a copy of the lockfile
+    $ conan create PkgB/1.1@user/testing --lockfile
+    # Now the lockfile has been modified, contains PkgB/1.1 instead of PkgB/1.0
+
+.. image:: ../images/lockfile_ci_1.png
+
+The next step is to know which dependants need to be built, because they are affected by the new
+**PkgB/1.1** version:
+
+.. code-block:: bash
+
+    $ conan graph build-order . --build=missing
+    [[PkgC, PkgD], [PkgE]]  # simplified format
+
+This command will return a list of lists, in order, of those packages to be built. We take the
+first sub-list and each package can be rebuilt, making sure the lockfile is applied:
+
+.. code-block:: bash
+
+    $ 
+
+
+.. image:: ../images/lockfile_ci_2.png