Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs for build_id & multi-config #161

Merged
merged 1 commit into from
Mar 1, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packaging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ This section shows how to create, build, upload and test your packages
packaging/creating_out
packaging/testing
packaging/upload
packaging/package_info
packaging/package_tools


Expand Down
190 changes: 190 additions & 0 deletions packaging/package_info.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
Package identifiers and information
====================================

Packages provide two types of information for consumers. The first one would be the package
identifiers (ID), which is a SHA-1 hash of the package configuration (settings, options, requirements),
that allows consumers to reuse an existing package without building it again from sources.

The other type of information would be C/C++ build information, as include paths, library names, or
compile flags
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks confusing to me. I wouldn't say that a package identifier is an information for me, the consumer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, it is printed in the conan info command, is useful information for consumers of the package, to know which binary they are actually using for a given configuration.

But, yes, I agree that the title/description could be improved.



Package IDs
------------

package_id
Copy link
Contributor

@lasote lasote Feb 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure about this file in the docs. Maybe all this should go to conanfile.py page and this page just explain the general why and link to conanfile.py page.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And maybe the conanfile.py page should be splitted in several subpages. It's becoming too big.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nop, the conanfile_py.rst if you meant that, is just for package consumers. This is only relevant for package creators, and I think it should go inside the "Creating packages section".

++++++++++

Each package has two elements that affect its ID: the configuration and the ``package_id()``
recipe method. The configuration is:

- The package settings, as specified in the recipe ``settings = "os", "compiler", ...``
- The package options, as specified in the recipe ``options = {"myoption": [True, False]}`` with
possible default values as ``default_options="myoption=True"``
- The package requirements, as defined in the package ``requires = "Pkg/0.1@user/channel"`` or in
the ``requirements()`` method.

Those elements are first declared in the recipe, but it is at install time when they get specific
values (as os=Windows, compiler=gcc, etc). Once all those elements have a value, they are hashed,
and a sha1 ID is computed. This will be the package identifier. It means that all configurations
that obtain the same ID, will be compatible.

For example, a header-only library with no dependencies will have no settings, options or requirements.
Hashing such empty items will obtain always the same ID, irrespective of os, compiler, etc. That is pretty
right, the final package for a header-only is just one package for all system, containing such headers.

If you need to change in some way that package compatibility, you can provide your own ``package_id()``
method, that can change the ``self.info`` object according to your compatibility model. For example,
a ``compiler.version`` setting is needed for building the package, which will be done with gcc-4.8.
But we know that such package will be compatible for gcc-4.9 for some reason (maybe just pure C code),
so we don't want to create a different binary for gcc-4.9 but let users with that configuration consume
the package created for gcc-4.8. You could do:

.. code-block:: python

from conans import ConanFile, CMake, tools
from conans.model.version import Version

class PkgConan(ConanFile):
name = "Pkg"
version = "0.1"
settings = "os", "compiler", "build_type", "arch"

def package_id(self):
v = Version(str(self.settings.compiler.version))
if self.settings.compiler == "gcc" and (v >= "4.8" and v < "5.0"):
self.info.settings.compiler.version = "gcc4.8/9"

Note that object being modified is ``self.info``. Also, any string is valid, as long as it will
be the same for the settings you want it to be the same package.

Read more about this in :ref:`how_to_define_abi_compatibility`

build_id
++++++++++
The ``build_id()`` methods is an optimization. If you find that you are doing exactly the same
build for two different packages, then you might want to use this method to redefine the build ID.
The build ID, by default is the same as the package ID, i.e. there is one build folder per package.
But if for any reason you have a build system that is building different artifacts in the same
build, and you want to create different packages with those artifacts, depending on different
settings, you don't want to rebuild again the same, as it it usually time consuming.

Read more about this in :ref:`build_id`

Package information
---------------------

Single configuration
+++++++++++++++++++++

A typical approach is to have each package contain the artifacts just for one configuration. In this
approach, for example, the debug pre-compiled libraries will be in a different package than the
release pre-compiled libraries.

In this approach, the ``package_info()`` method can just set the appropriate values for consumers,
to let them know about the package library names, and necessary definitions and compile flags.

.. code-block:: python

def package_info(self):
self.cpp_info.libs = ["mylib"]

The values that packages declare here (the ``include``, ``lib`` and ``bin`` subfolders are already
defined by default, so they define the include and library path to the package) are translated
to variables of the respective build system by the used generators. That is, if using the ``cmake``
generator, such above definition will be translated in ``conanbuildinfo.cmake`` to something like:

.. code-block:: cmake

set(CONAN_LIBS_MYPKG mylib)
...
set(CONAN_LIBS mylib ${CONAN_LIBS})

Those variables, will be used in the ``conan_basic_setup()`` macro to actually set cmake
relevant variables.


Read more about this in :ref:`package_info`

Multi configuration
+++++++++++++++++++++

It is possible that someone wants to package both debug and release artifacts in the same package,
so it can be consumed from IDEs like Visual Studio changing debug/release configuration from the IDE,
and not having to specify it in the command line.

Creating a multi-configuration Debug/Release package is not difficult, using ``CMake`` for example
could be:


.. code-block:: python

def build(self):
cmake = CMake(self.settings)
if cmake.is_multi_configuration:
cmd = 'cmake "%s" %s' % (self.conanfile_directory, cmake.command_line)
self.run(cmd)
self.run("cmake --build . --config Debug")
self.run("cmake --build . --config Release")
else:
for config in ("Debug", "Release"):
self.output.info("Building %s" % config)
self.run('cmake "%s" %s -DCMAKE_BUILD_TYPE=%s'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new configure/build helpers?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really. The helpers use the standard cmake.command_line which depend on `sèttings.build_type`` in this case not defined. So I am manually overriding the behavior. Probably it could be wrapped in another cmake helper method.

% (self.conanfile_directory, cmake.command_line, config))
self.run("cmake --build .")
shutil.rmtree("CMakeFiles")
os.remove("CMakeCache.txt")

In this case, we are assuming that the binaries will be differentiated with a suffix, in cmake syntax:

.. code-block:: cmake

set_target_properties(mylibrary PROPERTIES DEBUG_POSTFIX _d)


Such a package can define its information for consumers as:

.. code-block:: python

def package_info(self):
self.cpp_info.release.libs = ["mylibrary"]
self.cpp_info.debug.libs = ["mylibrary_d"]


This will translate to the cmake variables:

.. code-block:: cmake

set(CONAN_LIBS_MYPKG_DEBUG mylibrary_d)
set(CONAN_LIBS_MYPKG_RELEASE mylibrary)
...
set(CONAN_LIBS_DEBUG mylibrary_d ${CONAN_LIBS_DEBUG})
set(CONAN_LIBS_RELEASE mylibrary ${CONAN_LIBS_RELEASE})

And these variables will be correctly applied to each configuration by ``conan_basic_setup()``
helper.

In this case you can still use the general, no config-specific variables. For example, the
include directory, set by default to ``include`` is still the same for both debug and release.
Those general variables will be applied for all configurations.

Also, you can use any custom configuration you might want, they are not restricted. For example,
if your package is a multilibrary package, you could try doing something like:

.. code-block:: python

def package_info(self):
self.cpp_info.regex.libs = ["myregexlib1", "myregexlib2"]
self.cpp_info.filesystem.libs = ["myfilesystemlib"]

These specific config variables will not be automatically applied, but you can directly use them
in your consumer CMake build script.

.. note::

The automatic conversion of multi-config variables to generators is currently only implemented
in the ``cmake`` and ``txt`` generators. If you want to have support for them in another
build system, please open a github issue for it.



38 changes: 38 additions & 0 deletions reference/conanfile.rst
Original file line number Diff line number Diff line change
Expand Up @@ -558,6 +558,7 @@ And it will copy the lib to the package folder *lib/Mylib.lib*, which can be lin
might be able to reuse it for this ``package()`` method. Please check :ref:`reuse_cmake_install`


.. _package_info:

package_info()
---------------
Expand Down Expand Up @@ -819,6 +820,43 @@ But sometimes you would need to alter the general behavior, for example, to have

Please, check the section :ref:`how_to_define_abi_compatibility` to get more details.

.. _build_id:

build_id()
------------

In the general case, there is one build folder for each binary package, with the exact same hash/ID
of the package. However this behavior can be changed, there are a couple of scenarios that this might
be interesting:

- You have a build script that generates several different configurations at once, like both debug
and release artifacts, but you actually want to package and consume them separately. Same for
different architectures or any other setting
- You build just one configuration (like release), but you want to create different binary packages
for different consuming cases. For example, if you have created tests for the library in the build
step, you might want to create to package, one just containing the library for general usage, but
another one also containing the tests, as a reference and a tool to debug errors.

In both cases, if using different settings, the system will build twice (or more times) the same binaries,
just to produce a different final binary package. With the ``build_id()`` method this logic can be
changed. ``build_id()`` will create a new package ID/hash for the build folder, and you can define
the logic you want in it, for example:

.. code-block:: python

settings = "os", "compiler", "arch", "build_type"

def build_id(self):
self.info_build.settings.build_type = "Any"

So this recipe will generate a final different package for each debug/release configuration. But
as the ``build_id()`` will generate the same ID for any ``build_type``, then just one folder and
one build will be done. Such build should build both debug and release artifacts, and then the
``package()`` method should package them accordingly to the ``self.settings.build_type`` value.
Still different builds will be executed if using different compilers or architectures. This method
is basically an optimization of build time, avoiding multiple re-builds.



Other
------
Expand Down