Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add MakeDeps generator #14133

Merged
merged 56 commits into from
Aug 28, 2023
Merged

Conversation

uilianries
Copy link
Member

@uilianries uilianries commented Jun 20, 2023

Changelog: Feature: Add support to Makefile by the new MakeDeps generator
Docs: conan-io/docs#3348
closes #13394

This PR adds a new feature, the MakeDeps generator, that can be used with Conan install command: conan install --requires=foobar/0.1.0 -g MakeDeps. The idea is bringing back the make generator from Conan 1.x but with steroids, including support for components.
As real example, using Poco package, we will have:

% conan install --requires=poco/1.12.1 -g MakeDeps 
% ls *.mk
conandeps.mk

The are 3 levels of dependencies here:

  • Each Poco component and its dependencies will generate separated variables, that contains values related to that component, plus, dependencies;
  • Then, each package, include Poco, will have variable with all components listed, and all variables appending components;
  • As top level in conandeps.mk, all global variables like CONAN_LIB_DIRS, will list all dependencies.
  • Refer to the issue that supports this Pull Request.
  • If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
  • I've read the Contributing guide.
  • I've followed the PEP8 style guides for Python code.
  • I've opened another PR in the Conan docs repo to the develop branch, documenting this one.

Copy link
Member

@memsharded memsharded left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is looking good, please have a look at the comments.

Also, it would be better to focus on 1 good functional test that proves that this generator can actually be used to link with a regular library, and start from there to more advanced and complete cases (but that can happen later in other PRs)

conan/tools/gnu/makedeps.py Outdated Show resolved Hide resolved
conan/tools/gnu/makedeps.py Outdated Show resolved Hide resolved
conan/tools/gnu/makedeps.py Outdated Show resolved Hide resolved
conans/test/integration/toolchains/gnu/test_makedeps.py Outdated Show resolved Hide resolved
conans/test/integration/toolchains/gnu/test_makedeps.py Outdated Show resolved Hide resolved
@iskunk
Copy link
Contributor

iskunk commented Jun 20, 2023

Trying this out on my side. A few comments:

  • What's the motivation for breaking things out into various separate *.mk files? I haven't seen any other generators do that.

  • In the conandeps.mk file, I see a bit like

    # Global dependencies
    include libxslt.mk libxml2.mk zlib.mk libiconv.mk
    

    This assumes that the files are in the current directory, so if your project is doing e.g. include /path/to/conandeps.mk, Make will fail to find them.

  • Why the ubiquitous use of +=? Note that non-GNU Make programs may not support this operator. (I considered using this, perhtaps with a tool option to allow disabling them if needed, but then I didn't really see a point to it. Appending values is meaningfully different only if you have variable names colliding (which shouldn't happen) or if the consuming project sets a value on them first (I couldn't think of why they would want to do that).

@uilianries
Copy link
Member Author

@iskunk thank you for reviewing, your feedback is valuable.

What's the motivation for breaking things out into various separate *.mk files? I haven't seen any other generators do that.

Most of generators split in several files, including pkg-config and cmake, please, take a look in https://docs.conan.io/2/reference/tools/cmake/cmakedeps.html#generated-files for instance.

This assumes that the files are in the current directory, so if your project is doing e.g. include /path/to/conandeps.mk, Make will fail to find them.

Indeed it's a fragile point, need to be improved, thank you for spotting it.

Why the ubiquitous use of +=? Note that non-GNU Make programs may not support this operator. (I considered using this, perhtaps with a tool option to allow disabling them if needed, but then I didn't really see a point to it. Appending values is meaningfully different only if you have variable names colliding (which shouldn't happen) or if the consuming project sets a value on them first (I couldn't think of why they would want to do that).

GNU Make is by far, the most popular and well documented Make implementation, that's why the choice. Append variables is not expected but can occur, as users demand all kind of customization. Anyway, using only = should simplify the implementation even more, so it should it adapted.

@iskunk
Copy link
Contributor

iskunk commented Jun 21, 2023

Most of generators split in several files, including pkg-config and cmake, please, take a look in https://docs.conan.io/2/reference/tools/cmake/cmakedeps.html#generated-files for instance.

Ah, okay, now I remember those.

I'd point out that the CMake find_package() and pkg-config mechanisms both obtain information on a specific dependency by looking for a corresponding filename (e.g. FindFoo.cmake or foo.pc), so by necessity they have to break things out into separate files. That's not the case with plain Make, so I don't really see a benefit to taking that approach here.

GNU Make is by far, the most popular and well documented Make implementation, that's why the choice. Append variables is not expected but can occur, as users demand all kind of customization. Anyway, using only = should simplify the implementation even more, so it should it adapted.

One of my potential use cases is running a build on AIX or Solaris, and GNU Make isn't always available, so the greater compatibility will be a plus. (I forgot to point out the ?= and $(sort ...) bits as well. The sort was especially concerning since the order of at least CONAN_LIBS can be meaningful.)

@uilianries
Copy link
Member Author

uilianries commented Jun 22, 2023

@iskunk

I'd point out that the CMake find_package() and pkg-config mechanisms both obtain information on a specific dependency by looking for a corresponding filename (e.g. FindFoo.cmake or foo.pc), so by necessity they have to break things out into separate files. That's not the case with plain Make, so I don't really see a benefit to taking that approach here.

Yes, but keeping the flexibility of what you want to include is also a feature. For those cases when you want to use a separated folder to generate your makefiles, now you can use the definition CONAN_MAKEFILES_DIR when invoking make, so include directive will have a prefix with the folder name.

One of my potential use cases is running a build on AIX or Solaris, and GNU Make isn't always available, so the greater compatibility will be a plus. (I forgot to point out the ?= and $(sort ...) bits as well. The sort was especially concerning since the order of at least CONAN_LIBS can be meaningful.)

Those are very rare cases, most used for production, not development (I know some people using AIX). About the $(sort) you are right, need to be improved and will. The += and ?= were replaced by the simple =.

@iskunk
Copy link
Contributor

iskunk commented Jun 22, 2023

Yes, but keeping the flexibility of what you want to include is also a feature. For those cases when you want to use a separated folder to generate your makefiles, now you can use the definition CONAN_MAKEFILES_DIR when invoking make, so include directive will have a prefix with the folder name.

All these files do is assign variables. If there are some dependencies in there that I don't want to use, then I can just not use those variables in my makefile. What problem is solved by including foobar.mk instead of conandeps.mk?

I notice that the individual *.mk files are each including their own component dependencies, too, which means that many of these are being multiply included---by a large margin. I used your example of generating for poco/1.12.1, and added this line to the top of poco_foundation.mk:

$(info hello from poco_foundation.mk)

Each time GNU Make sees that, it will print out that message. I wrote a trivial makefile that includes conandeps.mk. When I run it, here is what I see:

$ make
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
hello from poco_foundation.mk
true

Also, CONAN_LIBS_POCO is the concatenation of all the component CONAN_LIBS_POCO_* variables, which is incorrect behavior. Whatever the package sets for its top-level cpp_info.libs field is what should show up there; the generator should not make up its own value in this manner. If it makes sense for a package to aggregate all its component libraries into the top-level component, then it will do so, in its package_info() method. (Same issue applies to the other variables; it is just particularly egregious for LIBS)

Those are very rare cases, most used for production, not development (I know some people using AIX).

My organization is using Conan to drive production builds of third-party dependencies. If all I cared about were development, we'd just use Conda.

The Conan buildenv mechanism is especially helpful on traditional Unix systems that can't run Python 3---I can run Conan on a Linux system, generate into a shared NFS workspace, have the Unix system load the buildenv and build in that NFS workspace, and then Conan on Linux can pick up the artifacts.

@iskunk
Copy link
Contributor

iskunk commented Jun 27, 2023

Much obliged @uilianries, this is looking pretty good.

I have a concern with how variables for components are named. In my implementation, I did a straight conversion from the foo::bar names, for example

  • boost::atomic --> CONAN_LIBS_BOOST_ATOMIC
  • pcre2::pcre2-8 --> CONAN_LIBS_PCRE2_PCRE2_8

You mentioned that you didn't like the "PCRE2_PCRE2" name-doubling, and I see your code detects and drops the redundant part of the name, e.g.

  • boost::atomic --> CONAN_LIBS_BOOST_ATOMIC
  • pcre2::pcre2-8 --> CONAN_LIBS_PCRE2_8

But say you have a package with component names like

  • foo::bar
  • foo::foo-bar

You'll get two CONAN_LIBS_FOO_BAR assignments. (This name combination may be unusual, but I don't believe it's prohibited)

The name-doubling from my implementation wasn't pretty, but it only reflected the redundant names in the original components. In the case of pcre2, the overall package is called pcre2, and the individual component libraries are named pcre2-8, pcre2-16, etc.---they are never referred to as 2-8, nor just 8, etc. There is no shorter, recognized name that can be given to them that would eliminate "pcre2" appearing twice in the package::component identifier.

(I have remarks on some other aspects of the current draft, but this is the main one for now)

@uilianries
Copy link
Member Author

@iskunk that's a nice feedback indeed, I didn't consider that scenario, because is rare, but still valid. I'll update the PR with the package name as prefix. Thank you!

@uilianries uilianries force-pushed the feature/make-generator branch from 27f9bee to 222a448 Compare June 27, 2023 14:48
Signed-off-by: Uilian Ries <uilianries@gmail.com>
@uilianries uilianries force-pushed the feature/make-generator branch from 222a448 to 842673c Compare June 27, 2023 15:04
Signed-off-by: Uilian Ries <uilianries@gmail.com>
@iskunk
Copy link
Contributor

iskunk commented Jun 28, 2023

All right, I gave the generator another test run. Here are my observations:

  • It's good to have the CONAN_HOST_* and CONAN_BUILD_* settings variables, but didn't @memsharded say those needed to go into a toolchain file, rather than the dependencies? (I'm assuming this means something like a separate conantoolchain.mk file, but am not clear on the distinction)

  • A couple things with a line like

    CONAN_SYSROOT_ZLIB = $(CONAN_ROOT_ZLIB)/.
    

    What's the /. at the end for? That is a redundant directory reference. (/tmp/foo, /tmp/foo/., and /tmp/foo/./././. all refer to the same directory)

    Secondly, why is the SYSROOT variable being set for a package that doesn't define it? (If you look at zlib's conanfile, there is no self.cpp_info.sysroot assignment, so this value is coming out of thin air)

  • I noticed the behavior of how (at least) CONAN_{LIBS,DEFINES,REQUIRES}_* are aggregated from the components if there is no value set at the top level. So for example, for pcre2/10.42, we get

    CONAN_LIBS_PCRE2 = \
          $(CONAN_LIBS_PCRE2_PCRE2_8) \
          $(CONAN_LIBS_PCRE2_PCRE2_POSIX) \
          $(CONAN_LIBS_PCRE2_PCRE2_16) \
          $(CONAN_LIBS_PCRE2_PCRE2_32)
    

    I don't see how this is helpful... this assumes that the consumer might want to use all the component libraries together, but that's not the case for a package like PCRE2. (Most applications, if they use PCRE2, use only one of those libraries.) Maybe it makes sense to do this for some packages, but then, those packages can always set things up this way explicitly in their conanfile. Setting an aggregated value like this in the generator has the potential to confuse the developer, as it suggests a way of using the package that is not correct.

    In general, I'd be very careful with having the generator "lie" about the value of package variables for the sake of convenience. Usually, it is better to fix up the conanfile to give those more convenient values.

  • If a package sets .defines at both the top and component levels, then the top-level CONAN_DEFINES_* variable is missing the $(CONAN_DEFINE_FLAG) prefix. (For example, try adding self.cpp_info.defines = "FOO" to the PCRE2 conanfile)

@uilianries
Copy link
Member Author

@iskunk Thank you for checking the implementation again. The patch is now much simpler, without jinja2.

What's the /. at the end for? That is a redundant directory reference. (/tmp/foo, /tmp/foo/., and /tmp/foo/./././. all refer to the same directory)
Secondly, why is the SYSROOT variable being set for a package that doesn't define it? (If you look at zlib's conanfile, there is no self.cpp_info.sysroot assignment, so this value is coming out of thin air)

The /. means current directory, but indeed is not needed. Indeed it's a bug that should be fixed, as sysroot is not assigned by default in cpp_info. It should be removed in a next commit. Thank you for spotting it.

It's good to have the CONAN_HOST_* and CONAN_BUILD_* settings variables, but didn't @memsharded say those needed to go into a toolchain file, rather than the dependencies? (I'm assuming this means something like a separate conantoolchain.mk file, but am not clear on the distinction)

Yes, splitting makes sense, as we have done for environment files in Conan 2.x

I don't see how this is helpful... this assumes that the consumer might want to use all the component libraries together, but that's not the case for a package like PCRE2. (Most applications, if they use PCRE2, use only one of those libraries.) Maybe it makes sense to do this for some packages, but then, those packages can always set things up this way explicitly in their conanfile. Setting an aggregated value like this in the generator has the potential to confuse the developer, as it suggests a way of using the package that is not correct.

If you want to use only one component, you pick only those variables related to that specific component. I don't see the scenario that you describe, maybe could write a test to validate the case, but I still see it as pretty rare case that would not worth covering now.

Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
@iskunk
Copy link
Contributor

iskunk commented Aug 14, 2023

@SSE4, there is no need for this generator to use GNU Make-specific features; all it does is define a bunch of variables. My original implementation was portable across different Make implementations, and I've ensured it remains that way.

@igagis
Copy link

igagis commented Aug 14, 2023

I'll just repost my question from other discussion , as it might be relevant here:

Why using custom CONAN_ variables as it was in conan 1? Why not go with same approach as in AutotoolsDeps? I.e. setting well known (by GNU make) variables like CXXFLAGS, LDFLAGS, LDLIBS in the MakeDeps generator?

In my opinion, adding MakeDeps generator would be just to fork the AutotoolsDeps generator, removing autotools related stuff from there and add setting LDLIBS variable instead of LIBS varaible. Not sure if it that easy, but what I'd expect from MakeDeps generator is just to set ASFLAGS, CFLAGS, CXXFLAGS, CPPFLAGS, LDFLAGS, LDLIBS varaibles, as listed in GNU make docs: https://www.gnu.org/software/make/manual/html_node/Implicit-Variables.html#index-CFLAGS. I'm not sure the generator has to set all the weird vars listed there, like PFLAGS...

Using these well known variables will result in that modifying makefiles of existing libraries will not be required at all to package them for conan

@iskunk
Copy link
Contributor

iskunk commented Aug 14, 2023

Hi @igagis,

There are several issues with the AutotoolsDeps approach using the *FLAGS variables, not least that (1) it assumes which variables are meaningful to the build system (what if you need to set LOCAL_CXX_FLAGS instead of CXXFLAGS? or pass them as a command-line arg instead of an env var?), and (2) there is no easy way for subsequent scripting to obtain flags for specific dependencies; it is all of them or nothing. The MakeDeps generator is intended to be fully general, in the way that Make itself is fully general. It sets CONAN_* variables to avoid collisions with ones already in use, and lets you make use of them however you like.

@reedhedges
Copy link

reedhedges commented Aug 14, 2023 via email

@igagis
Copy link

igagis commented Aug 15, 2023

Well, in addition to CONAN_ variables it could also set CFLAGS etc. variables, should not harm

@iskunk
Copy link
Contributor

iskunk commented Aug 15, 2023

Well, in addition to CONAN_ variables it could also set CFLAGS etc. variables, should not harm

Someone else would complain that they don't want the generator assigning to such variables---either because they want to set a different value, or because they don't want these set at all. (For example, I'm using the profile buildenv section to set CFLAGS, so I really don't want that showing up in the generator output.)

There is a bright line between "makes my use case easy to support" and "supports my use case out of the box." This generator opts for the former, because the latter is impossible to attain without a gazillion parameters.

@igagis
Copy link

igagis commented Aug 16, 2023

As far as I understand, different conan package means different conanfile.py. And which flags to add to CFLAGS etc. is defined by the conanfile.py.
Packaging a library for conan means writing conanfile.py for it and not modifying the library's build system files.

So, anyway, I'm just suggesting. Looks like I'll continue using AutotoolsDeps generator for my purposes, as it works pretty well, just needs one LDLIBS=LIBS small workaround.

conans/test/functional/toolchains/gnu/test_makedeps.py Outdated Show resolved Hide resolved
conans/test/integration/toolchains/gnu/test_makedeps.py Outdated Show resolved Hide resolved
conans/test/integration/toolchains/gnu/test_makedeps.py Outdated Show resolved Hide resolved
conans/test/integration/toolchains/gnu/test_makedeps.py Outdated Show resolved Hide resolved
conan/tools/gnu/makedeps.py Outdated Show resolved Hide resolved
Comment on lines +618 to +625
build_req = self._conanfile.dependencies.build # tool_requires
test_req = self._conanfile.dependencies.test

content_buffer = f"{self._title}\n"
deps_buffer = ""

# Filter the build_requires not activated for any requirement
dependencies = [tup for tup in list(host_req.items()) + list(build_req.items()) + list(test_req.items()) if not tup[0].build]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just drop build-requires if they are not going to be used at all?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exactly. We could add another feature, just like for pkgconfigdeps, the build_context_activated, a list with reference that should be part of the makefile.

Comment on lines 521 to 523
self._info.dirs = list(set(self._info.dirs))
self._info.flags = list(set(self._info.flags))
return self._info
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a weird pattern, with inputs and outputs all inside MakeInfo, seems this should be encapsulated and automatic in MakeInfo

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in the commit c5cd0a5


def _get_sysroot(self, root: str) -> list:
"""
Get the sysroot of the dependency. Sysroot is a list of directories, or a single directory
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far we haven't treated sysroot as multiple directories, maybe this would be better to align with others and treat it jointly with other generators behavior.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed is not usual having multiple sysroot, but only one when cross-building. TBH, it seems be a bug, because returning [''] seems be something mixed, it should be empty string, but then was converted to a list to keep same pattern as other cpp_info dirs

for var, prefix_var in _common_cppinfo_variables().items():
cppinfo_value = getattr(dependency.cpp_info, var)
# Use component cpp_info info when does not provide any value
if not cppinfo_value and hasattr(dependency.cpp_info, "components"):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cpp_info always has components defined, it is in the constructor of CppInfo, this hasattr check is useless?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doing some tests, it seems be. The idea here is capturing cpp_info value from a component, when a dependency has no such cpp_info value. It's available on the commit 614c273

uilianries and others added 7 commits August 21, 2023 10:42
Co-authored-by: James <memsharded@gmail.com>
Co-authored-by: James <memsharded@gmail.com>
Co-authored-by: James <memsharded@gmail.com>
Co-authored-by: James <memsharded@gmail.com>
Co-authored-by: James <memsharded@gmail.com>
Co-authored-by: James <memsharded@gmail.com>
Co-authored-by: James <memsharded@gmail.com>
@uilianries uilianries assigned uilianries and unassigned uilianries Aug 21, 2023
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
Signed-off-by: Uilian Ries <uilianries@gmail.com>
@iskunk
Copy link
Contributor

iskunk commented Aug 29, 2023

Thanks @memsharded, @uilianries. This has been a long time in coming.

I've pushed a PR (#14605) with some minor tweaks on top of this, to polish it off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[question] What's the intended way to work with makefiles without autotools in 2.0
6 participants