Feature: link editable packages

[jgsogo]

Changelog: Feature: Add experimental support for packages in editable mode
Docs: conan-io/docs#1009

• closes Consider PackageFolders that abstract layout for cache, workspaces #3559, closes [-e] Define cli syntax for packages in editable mode #3903

• Needs docs

• Consider Design Recommendation for Workspaces and Editable #4142: may override local file with shared configuration in cache. (Accepted, but priority will be local file first, then cache one)

• require_namespace, is this the name for the argument? it will raise when reading the file if some namespaces are found but not expected?

• do we want to allow pattern matching in the .ini file? do we need full conan-references in the .ini file?

• add an argument to the conan link command to allow setting a custom file with the information of paths (it will allow the user to skip the local repo file if it exists)

• File naming: decide naming for CONAN_PACKAGE_LAYOUT_FILE and EDITABLES_FILE; both, the variable name and the underlying file.

• The ConanFilePath issue (Add common method to get ConanFilePath from command line #4157), pass a args.target directly to the conan_api and try to build a path to a conanfile.py there, or resolve the path closer to the CLI and pass an absolute path to a conanfile.py? https://github.com/conan-io/conan/pull/4181/files/a12f4a53b9f0c9575f06a842245acdb716dc9980#r243259310

• Issue with revisions: https://github.com/conan-io/conan/pull/4181/files/a12f4a53b9f0c9575f06a842245acdb716dc9980#r245127450

Substitutes #4152

[jgsogo]

A couple of comments

[jgsogo]

There are still some open issues, all of them listed as checkboxes in the first comment. My opinion on them:

---

• require_namespace, is this the name for the argument? it will raise when reading the file if some namespaces are found but not expected?

Note.- We haven't agree if this information at the cache level will be in a single file, inside a profile,... but we are taking it for granted.

Paths in the cache-file can be applied to an specific package or to every one (more on this in the next issue), while paths in the repo-file can be applied only to the package where the file is. So, the repo-file wont have namespaces (sections like [pck_name:includedirs]) while the repo-file needs to know which package the section is referring to.

The question is: do we raise if we found a section with namespaces in the repo-file? do we raise if we found a section without namespaces in the cache-file?

My opinion: if it is a dedicated file I will raise.

---

• do we want to allow pattern matching in the .ini file?

In the cache-file, sections must be identified so we know which package they are referring to. At this stage of development, there are only two options:

• [*:includedirs]: applies to every package (lower priority)
• [pck_A:includedirs]: applies only to package pck_A (higher priority)

If we introduce some kind of package matching then sections like [pck_*:includedirs] will appear and we will need to agree on priority between different patterns (and we sure that the parser preserves this ordering)

---

• do we need full conan-references in the .ini file?

In the cache-file, now we are considering just the package name, does it make sense to include full conan reference?

---

• add an argument to the conan link command to allow setting a custom file with the information of paths (it will allow the user to skip the local repo file if it exists)

I would vote to merge this feature without that argument in the CLI, but I agree that it is the only way for the user to skip the local repo file if it exists in order to use the cache-file.

---

• File naming: decide naming for CONAN_PACKAGE_LAYOUT_FILE and EDITABLES_FILE; both, the variable name and the underlying file.

I will go for CONAN_PACKAGE_LAYOUT_FILE (.conan_package_layout) and LINKED_PACKAGES_LAYOUT_FILE (linked_packages_layout.ini).

Also, I would change LINKED_FOLDER_SENTINEL to LINKED_PACKAGE_SENTINEL

---

• The ConanFilePath issue (Add common method to get ConanFilePath from command line #4157), pass a args.target directly to the conan_api and try to build a path to a conanfile.py there, or resolve the path closer to the CLI and pass an absolute path to a conanfile.py? https://github.com/conan-io/conan/pull/4181/files/a12f4a53b9f0c9575f06a842245acdb716dc9980#r243259310

I think that the best way to do it is: build the absolute path to the conanfile.py in the Command class (it knows about the current working dir) and the conan_api will receive it.

---

• Issue with revisions: https://github.com/conan-io/conan/pull/4181/files/a12f4a53b9f0c9575f06a842245acdb716dc9980#r245127450

Needs @lasote feedback.

[lasote]

For me, the more aligned with the last comments would be to create a function in the conan API class or a function in the module while we define better how we want it. So, the command module can call that function and then pass the conanfile to the conan API method. I agree with @memsharded that the function to get the conanfile path doesn't belong to the command module.

[lasote]

The issue is not exactly with the revisions, assign there a revision fix it by chance. In the info_on_child_test there is a node in the graph that is the "conanfile.py", that node has None/None@None/None reference. So, comparing with this code in the graph.py crashes because it doesn't early return:

    def __cmp__(self, other):
        if other is None:
            return -1
        elif self.conan_ref is None:
            return 0 if other.conan_ref is None else -1
        elif other.conan_ref is None:
            return 1

        if self.conan_ref == other.conan_ref:
            return 0

        # Cannot compare None with str
        if self.conan_ref.revision is None and other.conan_ref.revision is not None:
            return 1

        if self.conan_ref.revision is not None and other.conan_ref.revision is None:
            return -1

        if self.conan_ref < other.conan_ref:
            return -1
        return 1

Said that, @memsharded @jgsogo confirm that the None/None@None/None reference is intended, I assume that a better comparison code should be introduce for that case. Probably a unit test case would be necessary.

[lasote]

I'm ok if at this point we only allow "*"

[lasote]

About the "general" file to declare layouts. I would like to discuss together the Jerry idea about having profiles for that.
As he said, from next release profile can be composable, that is, specify several profiles at the same time.
So, what if I declare a profile only defining editables stuff? I could have any number of profiles for any reason, not only an editables_packages.ini, like profiles/visual_old_projects.layout or profiles/linux_common_editables or whatever.

[jgsogo]

If this is possible and those files/profile will contain sections listing editable_packages directories then we need to decide between two behaviors:

• the last section found with the same name overrides the previous ones
• each section will accumulate all the paths listed on every file.

This affects also to the criteria followed to get paths from the editables_packages.ini file (if we allow pattern matching we can come across several sections suitable for the same package)

[jgsogo]

There are just a couple of open questions yet, please one last effort and we can start thinking about merging it 💪

[lasote]

@jgsogo the docs are not linked in the body.

[danimtb]

Added docs! 💪