Filter directories

[memsharded]

Proposed performance boost and fixed tests, for #1711

Implements #1607

[codecov-io]

Codecov Report

Merging #1719 into develop will increase coverage by <.01%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop    #1719      +/-   ##
===========================================
+ Coverage    95.78%   95.78%   +<.01%     
===========================================
  Files          309      309              
  Lines        26196    26226      +30     
===========================================
+ Hits         25091    25121      +30     
  Misses        1105     1105

[solvingj]

For the record, pasted the new build_info into my dev environment and it did solve my problems. Thanks all

[memsharded]

Excellent! Thanks for testing and thanks all for the good job

[skizzay]

I just upgraded to 0.27.0 and noticed a regression that seems to be related to this change. I'm noticing that my CONAN_INCLUDE_DIRS_NIAGARA is always empty now if I specify more than one generator.

I've traced it down to conans.client.generators.cmake.DepsCppCmake. The constructor is accessing cpp_info.include_paths. However, this value was accessed by another generator previously before the directory was generated. Now, the cache is invalid at this point because the include_paths doesn't look to see if the directory exists.

We should have a way to purge the cache.

[memsharded]

I have created this test, and it is passing:

    def regression_test(self):
        # Possible regression of: https://github.com/conan-io/conan/pull/1719#issuecomment-339137460
        client = TestClient()
        client.save({"conanfile.py": """from conans import ConanFile
class Pkg(ConanFile):
    exports_sources = "*"
    def package(self):
        self.copy("*", dst="include")
""", "header.h": ""})
        client.run("create Pkg/0.1@user/stable")
        client.save({"conanfile.txt": "[requires]\nPkg/0.1@user/stable\n"
                     "[generators]\ncmake\ntxt\ngcc\n"}, clean_first=True)
        client.run("install .")

        cmake = load(os.path.join(client.current_folder, "conanbuildinfo.cmake"))
        txt = load(os.path.join(client.current_folder, "conanbuildinfo.txt"))
        gcc = load(os.path.join(client.current_folder, "conanbuildinfo.gcc"))

        self.assertIn("Pkg/0.1/user/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include",
                      cmake)
        self.assertIn("Pkg/0.1/user/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include",
                      txt)
        self.assertIn("Pkg/0.1/user/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include",
                      gcc)

Your issue might be related to some generator that is actually creating the folder? The package folder should be inmutable, that is why the cached value for the existence of include_paths should be valid for all generators. Generators have to be read-only they are executed the last step, after the package and the package manifest is created, so if they modify the package, they can break the package manifest. What is your use case for generating folders from a generator?

[skizzay]

It's the cmake generator, when it's generating conanbuildinfo.cmake via conan test_package skizzay/testing. From what I see in the package itself, everything looks good. The problem is the generated conanbuildinfo.cmake file not properly setting CONAN_INCLUDE_DIRS_PKG. The value is empty. In the test case above what is the value CONAN_INCLUDE_DIRS_PKG?

If you look at conans/client/generators/cmake.py on line 14:

self.include_paths = multiline(cpp_info.include_paths)

cpp_info.include_paths is empty at this point. However, if you change it to access cpp_info.includedirs,

self.include_paths = multiline(cpp_info.includedirs)

then we get the expected generated output. This is an indication that include_paths is being invoked before the paths are generated. As such, the resulting values are not being cached; thus, never returned when invoking include_paths.

[memsharded]

The value is empty. In the test case above what is the value CONAN_INCLUDE_DIRS_PKG?

It is pointing to the include path, that appears in the test check: Pkg/0.1/user/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include

cpp_info.include_paths is not a variable, it is a property implemented as:

    @property
    def include_paths(self):
        if self._include_paths is None:
            self._include_paths = self._filter_paths(self.includedirs)
        return self._include_paths

It will returns includedirs if the directories defined there actually exist, and it seems to work well here. I cannot find any place where it could be called before the package folders and layout (including "include" subfolders) is already there.

Maybe there is something in the other generators you are using. Which are they? Can you check from the output in which order are they being used? Does it happen with any other generator? Just cmake one? Thanks!

[skizzay]

My generators are "cmake", "ycm". I'm going to investigate this further and create a test case to reproduce. Where did you place that test case? I want to use that as a starting point.

[memsharded]

My branch: https://github.com/memsharded/conan/tree/feature/test_regression_filter_directories -> conans/test/generators/generator_filter_error_test.py

Let me know how it goes. Release 0.28 is tomorrow, branch already open, but if it indeed a bug, will try to include fix. Thanks!

[memsharded]

Hi @skizzay

Any news on this? We are preparing 0.29, would be nice to know if something is failing here. Thanks!

[memsharded]

I have added a test for this: #2074

It looks good, so if it is failing in your case, please provide a bit more info. Thanks!