[bug] The CMakeDeps generation is very slow

[fredizzimo]

Environment details

• Conan version: I'm using 1.59, but Conan 2.0 has the same issue.
• Python version: 3.6.10

Steps to reproduce

1. Create a conanfile with several dependencies
2. run conan install
3. Observe that the generate step of each package takes a long time.

For example in one of our smaller projects, the install command took 23.6 seconds. And profiling using the cProfile revealed that 21.4 seconds of that was spent compiling Jinja2 templates. The actual rendering took just a fraction of a second. This slowness really hurts our iteration speed, and for bigger projects it takes much longer.

The root cause is wrong usage of Jinja2, you are supposed to create an environment with a loader, and load all files through that environment. That way Jinja2 will automatically cache the results.

I was considering making a pull request, but then I noticed that Jinja2 is used in the same wrong way all our the place in the codebase, so there need to be some bigger architectural consideration on how to pass the environment around, or perhaps use some kind of singleton. All templates also need to be given an unique name or stored in files, so that the Jinja2 environment uniquely can identify them.

Logs

No response

[memsharded]

Hi @fredizzimo

Thanks for your report.

Quick question, how many dependencies are you talking about? For "small" projects, with just a few dozens of dependencies, it takes just a couple of seconds (33 deps = 3 seconds, for example). Extrapolating that, it seems your small project could have around +200 dependencies?

[fredizzimo]

Well, the main conanfile has 20 packages, with each of them having a few external dependencies and also depending on each other, in total there are 36 unique packages. If I count all the Required By from conan info I get count 60 packages. But the CMakeDeps are generated for indirect dependencies as well. So I did this

❯ find . -name "*config.cmake" | wc -l
130
❯ find . -name "Find*.cmake" | wc -l
43

I guess you estimation of 200 dependencies is not that far off from the truth.

[memsharded]

Ok, that makes sense, thanks for the feedback.

I think we can move this forward locally, without needing to optimize at the whole application level.
I'd start optimizing the CMakeDeps generation, and the bottleneck is probably:

 def render(self):
        try:
            context = self.context
        except Exception as e:
            raise ConanException("error generating context for '{}': {}".format(self.conanfile, e))
        if context is None:
            return
        return Template(self.template, trim_blocks=True, lstrip_blocks=True,
                        undefined=jinja2.StrictUndefined).render(context)

It sounds that by just caching the Template(self.template, trim_blocks=True, lstrip_blocks=True, undefined=jinja2.StrictUndefined) for each different file type (xxx-data.cmake, xxx-config.cmake, xxx-config-version.cmake, etc), we could have very large wins.

If not enough, then we could consider a loader at the CMakeDeps level, but lets go step by step. What do you think?

[fredizzimo]

Yes. that's the worst bottleneck.

I think caching the temlates should have the same effect as using a Jinja session and might be easier to implement than my initial idea, so it sounds like a good first appoach.

The idea was to add a class variable storing the session, and then give each of the template classes a new property template_name for example. That way a custom loader could easily be written, and you would request the template by it's name.

That could then possibly be furter extended to register all the Jinja templates, not only the CMakeDeps ones. The CMakeToolchain for example, probably also costs a few seconds for big projects with a lot of packages.

[memsharded]

I think caching the temlates should have the same effect as using a Jinja session and might be easier to implement than my initial idea, so it sounds like a good first appoach.

Good, lets try that. Do you want to give it a try?

That could then possibly be furter extended to register all the Jinja templates, not only the CMakeDeps ones. The CMakeToolchain for example, probably also costs a few seconds for big projects with a lot of packages.

Probably not, as the toolchain is rendering just 1 file conan_toolchain.cmake, if it has hundreds of dependencies, it will be just one loop over them to concatenate some paths, that will be rendered in that 1 file, but probably optimizing Jinja loading doesn't have any effect.

[fredizzimo]

Good, lets try that. Do you want to give it a try?

Yes, I can try to do it, that's probably the fastest way to get the fix available for us.

[jcar87]

Perhaps unrelated, but we had a similar issue reported not a long time ago and I believe it turned out a non-optimised version of the Python interpreter was running. Is there any chance you are building the python interpreter from source? Otherwise if you could provide more information about OS, architecture, and where the Python interpreter is coming from, we can check on our end.

I build recipes in Conan Center with similar numbers of dependencies and have not seen the conan install step take as long as 20+ seconds, it's usually a lot faster than that (even on a modest x86_64 machine). It could be a different story if one was running this, say, natively on a Raspberry Pi for instance, and there may indeed be things to improve, but I'm very surprised about these times, so any more info is appreciated.

[fredizzimo]

I created a pull request #13857

@jcar87, No, I have seen this issue with several different interpreters, both on Linux and Windows, and all of them have been official releases.

I also have a quite fast computer with a Xenon W-2135 processor, so it's not that.

But the problem is quite hard to spot, since you will only see it as the generate step for each package taking longer, and that can easily be hidden by if there's some additional compilation going on. But it's quite noticeable for us, with incremental compilation and our own "workspace" implementation. As a part of the cmake cache generation we re-install all the packages in the workspace in editable mode, to guarantee that any changes to a conanfile is taken into account.

I tested with a little bit bigger project with around 80 dependencies total and around 30 packages of it's own, and my fix saves around 1 minute, from 1:10 to 10 seconds.

[fredizzimo]

@memsharded, would it be possible to get this backported to Conan 1.XX? We are still using that, but hopefully for not too long anymore.

I apologize if it's already on the radar since it was probably too early to make it into the 1.60 release, but I don't see any 1.XX milestones assigned to this., so it might be forgotten There's also no hurry to make a release just for this.

[memsharded]

It was not in our plans. Last 1.60 we merged some apparently innocent PR, and it was a regression, and now we have to release 1.60.1 (I still need to get some time to help the users to add some unit tests, we haven't managed with so many ongoing things, and meanwhile some users are not very happy, etc.). So typically, unless there is something very critical, or necessary for the 1.X<->2.0 syntax compatibility, we prefer not to add changes to the 1.X releases.