Pkg-lists

common/experimental_warning.inc

@@ -1,2 +1,4 @@
 .. warning::
+
     This feature is experimental and subject to breaking changes.
+    See :ref:`the Conan stability<stability>` section for more information.

examples.rst

@@ -14,3 +14,4 @@ Examples
    examples/config_files
    examples/graph
    examples/dev_flow
+   examples/commands

examples/commands.rst

@@ -0,0 +1,9 @@
+.. _examples_commands:
+
+Conan commands examples
+=========================
+
+.. toctree::
+   :maxdepth: 2
+
+   commands/pkglists

examples/commands/pkglists.rst

@@ -0,0 +1,176 @@
+.. _examples_commands_pkglists:
+
+Using packages-lists
+====================
+
+.. include:: ../../common/experimental_warning.inc
+
+
+Packages lists are a powerful and convenient Conan feature that allows to automate and concatenate different Conan commands.
+Let's see some common use cases:
+
+
+Listing packages and downloading them
+-------------------------------------
+
+A first simple use case could be listing some recipes and/or binaries in a server, and then downloading them.
+
+We can do any ``conan list``, for example, to list all ``zlib`` versions above ``1.2.11``, the latest recipe revision,
+all Windows binaries for that latest recipe revision, and finally the latest package revision for every binary.
+Note that if we want to actually download something later, it is necessary to specify the ``latest`` package revision,
+otherwise only the recipes will be downloaded.
+
+.. code-block:: bash
+
+    $ conan list "zlib/[>1.2.11]#latest:*#latest" -p os=Windows --format=json -r=conancenter > pkglist.json
+
+
+The output of the command is sent in ``json`` format to the file ``pkglist.json`` that looks like:
+
+
+.. code-block:: json
+    :caption: pkglist.json (simplified)
+
+    "conancenter": {
+        "zlib/1.2.12": {
+            "revisions": {
+                "b1fd071d8a2234a488b3ff74a3526f81": {
+                    "timestamp": 1667396813.987,
+                    "packages": {
+                        "ae9eaf478e918e6470fe64a4d8d4d9552b0b3606": {
+                            "revisions": {
+                                "19808a47de859c2408ffcf8e5df1fdaf": {
+                                }
+                            },
+                            "info": {
+                                "settings": {
+                                    "arch": "x86_64",
+                                    "os": "Windows"
+                                }
+                            }
+                        }
+                    }
+                }
+            },
+        "zlib/1.2.13": {
+        }
+    }
+
+
+The first level in the ``pkglist.json`` is the "origin" remote or "Local Cache" if the list happens in the cache. 
+In this case, as we listed the packages in ``conancenter`` remote, that will be the origin.
+
+
+We can now do a download of these recipes and binaries with a single ``conan download`` invocation:
+
+.. code-block:: bash
+
+    $ conan download --list=pkglist.json -r=conancenter
+    # Download the recipes and binaries in pkglist.json
+    # And displays a report of the downloaded things
+
+
+Downloading from one remote and uploading to a different remote
+---------------------------------------------------------------
+
+Let's say that we create a new package list from the packages downloaded in the previous step:
+
+.. code-block:: bash
+
+    $ conan download --list=pkglist.json -r=conancenter --format=json > downloaded.json
+    # Download the recipes and binaries in pkglist.json
+    # And stores the result in "downloaded.json"
+
+
+The resulting ``downloaded.json`` will be almost the same as the ``pkglist.json`` file, but in this case, the "origin" of
+those packages is the ``"Local Cache"`` (as the downloaded packages will be in the cache):
+
+
+.. code-block:: json
+    :caption: downloaded.json (simplified)
+
+    "Local Cache": {
+            "zlib/1.2.12": {
+                "revisions": {
+                }
+            }
+        }
+
+That means that we can now upload this same set of recipes and binaries to a different remote:
+
+.. code-block:: bash
+
+    $ conan upload --list=downloaded.json -r=myremote -c
+    # Upload those artifacts to the same remote
+
+
+.. note::
+
+    **Best practices**
+
+    This would be a **slow** mechanism to run promotions between different server repositories. Servers like
+    Artifactory provide ways to directly copy packages from one repository to another without using a client, 
+    that are orders of magnitude faster because of file deduplication, so that would be the recommended approach.
+    The presented approach in this section might be used for air-gapped environments and other situations in which
+    it is not possible to do a server-to-server copy.
+
+
+
+Building and uploading packages
+-------------------------------
+
+One of the most interesting flows is the one when some packages are being built in the local cache, with a 
+``conan create`` or ``conan install --build=xxx`` command. Typically, we would like to upload the locally built
+packages to the server, so they don't have to be re-built again by others. But we might want to upload only
+the built binaries, but not all others transitive dependencies, or other packages that we had previously in
+our local cache.
+
+It is possible to compute a package list from the output of a ``conan install``, ``conan create`` and ``conan graph info``
+commands. Then, that package list can be used for the upload. Step by step:
+
+First let's say that we have our own package ``mypkg/0.1`` and we create it:
+
+.. code-block:: bash
+
+    $ conan new cmake_lib -d name=mypkg -d version=0.1
+    $ conan create . --format=json > create.json
+
+
+This will create a json representation of the graph, with information of what packages have been built ``"binary": "Build"``:
+
+.. code-block:: json
+    :caption: create.json (simplified)
+
+    {
+    "graph": {
+        "nodes": {
+            "0": {
+                "ref": "conanfile",
+                "id": "0",
+                "recipe": "Cli",
+                "context": "host",
+                "test": false
+            },
+            "1": {
+                "ref": "mypkg/0.1#f57cc9a1824f47af2f52df0dbdd440f6",
+                "id": "1",
+                "recipe": "Cache",
+                "package_id": "2401fa1d188d289bb25c37cfa3317e13e377a351",
+                "prev": "75f44d989175c05bc4be2399edc63091",
+                "build_id": null,
+                "binary": "Build"
+            }
+        }
+    }
+
+
+We can compute a package list from this file, and then upload those artifacts to the server with:
+
+.. code-block:: bash
+
+    $ conan list --graph=create.json --graph-binaries=build --format=json > pkglist.json
+    # Create a pkglist.json with the known list of recipes and binaries built from sources
+    $ conan upload --list=pkglist.json -r=myremote -c
+
+
+For more information see the :ref:`Reference commands section<reference_commands>`

introduction.rst

@@ -88,7 +88,7 @@ There are some things that are not included in this commitment:
 - Builtin default implementation of extension points as plugins or hooks can also change with every release. Users can provide their own ones for stability.
 - Output of packages templates with ``conan new`` can update at any time to use latest features.
 - The output streams stdout, stderr, i.e. the terminal output can change at any time. Do not parse the terminal output for automation.
-- Anything that is explicitily labeled as ``experimental`` or ``preview`` in the documentation, or in the Conan cli output.
+- Anything that is explicitly labeled as ``experimental`` or ``preview`` in the documentation, or in the Conan cli output. Read the section below for a detailed definition of these labels.
 - Anything that is labeled as ``deprecated`` in the documentation should not get new usages, as it will not get new fixes and it will be removed in the next major version.
 - Other tools and repositories outside of the Conan client
 

reference/commands/download.rst

@@ -6,34 +6,39 @@ conan download
 .. code-block:: text
 
     $ conan download -h
-    usage: conan download [-h] [-v [V]] [--only-recipe]
-                          [-p PACKAGE_QUERY] -r REMOTE
-                          reference
+    usage: conan download [-h] [-v [V]] [-f FORMAT] [--only-recipe]
+                          [-p PACKAGE_QUERY] -r REMOTE [-l LIST]
+                          [pattern]
 
     Download (without installing) a single conan package from a remote server.
 
     It downloads just the package, but not its transitive dependencies, and it will not call
     any generate, generators or deployers.
-    It can download multiple packages if patterns are used, and also works with queries over
-    the package binaries.
+    It can download multiple packages if patterns are used, and also works with
+    queries over the package binaries.
 
     positional arguments:
-      reference             Recipe reference or package reference, can contain *
-                            as wildcard at any reference field. If revision is not
-                            specified, it is assumed latest one.
+      pattern               A pattern in the form
+                            'pkg/version#revision:package_id#revision', e.g:
+                            zlib/1.2.13:* means all binaries for zlib/1.2.13. If
+                            revision is not specified, it is assumed latest one.
 
     optional arguments:
       -h, --help            show this help message and exit
       -v [V]                Level of detail of the output. Valid options from less
                             verbose to more verbose: -vquiet, -verror, -vwarning,
                             -vnotice, -vstatus, -v or -vverbose, -vv or -vdebug,
                             -vvv or -vtrace
+      -f FORMAT, --format FORMAT
+                            Select the output format: json
       --only-recipe         Download only the recipe/s, not the binary packages.
       -p PACKAGE_QUERY, --package-query PACKAGE_QUERY
                             Only download packages matching a specific query. e.g:
                             os=Windows AND (arch=x86 OR compiler=gcc)
       -r REMOTE, --remote REMOTE
                             Download from this specific remote
+      -l LIST, --list LIST  Package list file
+
 
 
 Downloads recipe and binaries to the local cache from the specified remote.
@@ -44,6 +49,15 @@ Downloads recipe and binaries to the local cache from the specified remote.
     download any of the transitive dependencies of the downloaded package.
 
 
+The ``conan download`` command can downlaod packages to 1 server repository specified by the ``-r=myremote`` argument.
+
+It has 2 possible and mutually exclusive inputs:
+
+- The ``conan download <pattern>`` pattern-based matching of recipes, with a pattern similar to the ``conan list <pattern>``.
+- The ``conan download --list=<pkglist>`` that will upload the artifacts specified in the ``pkglist`` json file
+
+
+
 You can use patterns to download specific references just like in other commands like
 :command:`conan list` (see the patterns documentation there :ref:`reference_commands_list`) or :command:`conan upload`:
 

reference/commands/list.rst

@@ -6,32 +6,39 @@ conan list
 .. code-block:: text
 
     $ conan list -h
-    usage: conan list [-h] [-f FORMAT] [-v [V]] [-p PACKAGE_QUERY]
-                      [-r REMOTE] [-c]
-                      reference
+    usage: conan list [-h] [-v [V]] [-f FORMAT] [-p PACKAGE_QUERY] [-r REMOTE]
+                  [-c] [-g GRAPH] [-gb GRAPH_BINARIES] [-gr GRAPH_RECIPES]
+                  [pattern]
 
     List existing recipes, revisions, or packages in the cache (by default) or the remotes.
 
     positional arguments:
-      reference             Recipe reference or package reference. Both can
-                            contain * as wildcard at any reference field. If
+      pattern               A pattern in the form
+                            'pkg/version#revision:package_id#revision', e.g:
+                            zlib/1.2.13:* means all binaries for zlib/1.2.13. If
                             revision is not specified, it is assumed latest one.
 
     optional arguments:
       -h, --help            show this help message and exit
-      -f FORMAT, --format FORMAT
-                            Select the output format: json, html
       -v [V]                Level of detail of the output. Valid options from less
                             verbose to more verbose: -vquiet, -verror, -vwarning,
                             -vnotice, -vstatus, -v or -vverbose, -vv or -vdebug,
                             -vvv or -vtrace
+      -f FORMAT, --format FORMAT
+                            Select the output format: json, html
       -p PACKAGE_QUERY, --package-query PACKAGE_QUERY
                             List only the packages matching a specific query, e.g,
                             os=Windows AND (arch=x86 OR compiler=gcc)
       -r REMOTE, --remote REMOTE
                             Remote names. Accepts wildcards ('*' means all the
                             remotes available)
       -c, --cache           Search in the local cache
+      -g GRAPH, --graph GRAPH
+                            Graph json file
+      -gb GRAPH_BINARIES, --graph-binaries GRAPH_BINARIES
+                            Which binaries are listed
+      -gr GRAPH_RECIPES, --graph-recipes GRAPH_RECIPES
+                            Which recipes are listed
 
 The ``conan list`` command can list recipes and packages from the local cache, from the
 specified remotes or from both. This command uses a *reference pattern* as input. The
@@ -309,8 +316,20 @@ To list all the package revisions for for the latest recipe revision:
     equivalent to ``zlib/1.2.11#latest:*#*``
 
 
-List json output
-----------------
+Listing graph artifacts
+-----------------------
+
+When the ``conan list --graph=<graph.json>`` graph json file is provided, the command will list the binaries in it.
+By default, it will list all recipes and binaries included in the dependency graph. But the ``--graph-recipes=<recipe-mode>``
+and ``--graph-binaries=<binary-mode>`` allow specifying what artifacts have to be listed in the final result, some examples:
+
+- ``conan list --graph=graph.json --graph-binaries=build`` list exclusively the recipes and binaries that have been built from sources
+- ``conan list --graph=graph.json --graph-recipes=*`` list exclusively the recipes, all recipes, but no binaries
+- ``conan list --graph=graph.json --graph-binaries=download`` list exclusively the binaries that have been downloaded in the last ``conan create`` or ``conan install``
+
+
+List json output format
+-----------------------
 
 .. note::
 
@@ -366,8 +385,9 @@ with the following structure:
     }
   }
 
-List html output
-----------------
+
+List html output format
+-----------------------
 
 The ``conan list ... --format=html`` will return a html output in ``stdout`` (which can be redirected to a file)
 with the following structure: