ivy_utils.mapjars() can be slow when invoked from ivy_resolve.execute()

[ericzundel]

I'm sorry I haven't had time to delve further. I wanted to capture this before I forgot about it because we've already changed our logic to avoid the problem. My guess is that there is some code that is dead and should be removed.

We ran across this yesterday when working on resurrecting pants goal idea. We added some code that called self.context.products.require('jar_dependencies') elsewhere that would trigger the inner loop in ivy_resolve.execute():

 create_jardeps_for = self.context.products.isrequired('jar_dependencies')
    if create_jardeps_for:
      genmap = self.context.products.get('jar_dependencies')
      for target in filter(create_jardeps_for, targets):
        self._ivy_utils.mapjars(genmap, target, executor=executor,
                                workunit_factory=self.context.new_workunit)

The result is that ivy is invoked many many times and its extremely slow... It takes minutes to bring up the IDE on some of our projects using this scheme because we have a lot of external dependencies.

Now, we observed that this code doesn't get called during a normal java compile. It appears that instead a single ivy.xml file is created and all these dependencies are resolved in a single invocation of ivy. There is a step named 'map_jars' from jvm_dependency_analyzer.py that prints out. We ended up analyzing the way that works and started looking in 'ivy_jar_products' and got the mapping that way.

• should anyone be using ivy_utils.mapjars() ? If not, let's nix it.
• if it is still useful, maybe we should change it ivy_resolve.execut() from using an inner loop that gets invoked many times to batching up the work and invoking it once.

[ericzundel]

I put in a fix for this in ivy_imports.py in commit 1aaf3b6 but had to back it out. See https://rbcommons.com/s/twitter/r/1364/

[ericzundel]

I think this is the gist of the problem described in issue #6

[ericzundel]

mapjars is now dead in a massive Ivy refactoring that jsirois did this past week.