[question] Metadata feature with conan build -of

[maitrey]

What is your question?

Dear Conan Folks,

I am using the following command for building packages:

conan build . -of output_folder --deployer=full_deploy --update --user autosar --channel featuretest -pr:b=default -pr:h=mdp

And I export the contents of the package afterwards.
When I use the metadata feature : https://docs.conan.io/2/devops/metadata.html
I donot use hooks but mainly using the copy calls in build and export method.
In the export method , I donot have access to self.build_folder or self.package_metadata_folder.
Is there any variable that can directly link to self.package_metadata_folder in the export method?
I have some test reports that I like to export as a metadata to the package.

Thanks a lot for your help!

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @maitrey

Thanks for your question

n the export method , I donot have access to self.build_folder or self.package_metadata_folder.

This is expected. At export time, there is no profile, configuration at all, so the package_id cannot be computed. The build and package folders only exist for a given configuration. You can check that the conan export command do not have settings or profile inputs.

The only metadata that can be saved at export time is recipe metadata, that is common for all the different package binary configurations.

Please let me know if this clarifies the issue.

[maitrey]

Thanks for getting back to me. I understand then I could copy to the recipe folder in CMakeLists or the build method and then use it in export method.
I would have one another question: Is it allowed to use Python-3.12 together with conan-2.0.11 in Windows context? Are there any known issues?

[memsharded]

Thanks for getting back to me. I understand then I could copy to the recipe folder in CMakeLists or the build method and then use it in export method.

No, that is not possible. build() happens always after export. So whatever it tries to put in the recipe metadata shouldn't be used by the recipe later. Also, if you check the docs, the metadata is not intended to be "used" by recipe, only saved. If your recipe relies on reading metadata, then it is not metadata, it should be regular export/package files.

I would have one another question: Is it allowed to use Python-3.12 together with conan-2.0.11 in Windows context? Are there any known issues?

There shouldn't be any issues for running with Python 3.12, but let us know if otherwise. Still always recommended to move to the latest patch for any minor version, the current patch is 2.0.13, please try to update when possible.

[maitrey]

I think I did not understand then the package metadata feature properly.
So build method looks like this for me:

  def build(self):
        cmake = CMake(self)
        cmake.configure()
        
        cmake.build()
        self.run(f"ctest --output-on-failure *")
        copy(self, "CompManager_Report.zip", src=os.path.join(self.source_folder, "report", "results"),
                dst=os.path.join(self.package_metadata_folder, "tests"))
        copy(self, "Klocwork*.html", src=os.path.join(self.build_folder, "Klocwork", "reports"),
                dst=os.path.join(self.package_metadata_folder, "tests"), excludes="data")

And the export method looks like this:

def export(self):
	copy(self, "CompManager_Report.zip", src=os.path.join(self.recipe_folder, "report", "results"),
			dst=os.path.join(self.recipe_metadata_folder, "tests"))
        copy(self, "Klocwork*.html", src=self.recipe_folder,
			dst=os.path.join(self.recipe_metadata_folder, "tests"), excludes="data")

If I donot use it in the export method then I do not see the needed metadata in the conan cache.
After conan build I do see a metadata folder in the self.build_folder with the needed data.
I confirm it by checking using:
conan cache path compmgr/1.0.0@autosar/release --folder=metadata
Without the export method, I see no contents in the metadata folder.

[memsharded]

The conan cache path compmgr/1.0.0@autosar/release --folder=metadata is the recipe metadata
To see the package metadata, you need to provide the package_id of the specific package you want to check (you can have multiple package binaries):

$ conan cache path compmgr/1.0.0@autosar/release:<package_id> --folder=metadata

As it seems you are putting there tests results, it doesnt have sense to put that as "recipe metadata", because the tests are build and executed after the recipe is exported/created, the tests only exists after build, so the tests results can only be package metadata, not recipe metadata.

[maitrey]

Thanks james.And do I use the build and export method correctly?

[maitrey]

When I run this example with conan build . -of testing

import os
from conan import ConanFile
from conan.tools.files import save, copy

class Pkg(ConanFile):
   name = "pkg"
   version = "0.1"

   def layout(self):
      # Or something else, like the "cmake_layout(self)" built-in layout
      self.folders.build = "mybuild"
      self.folders.generators = "mybuild/generators"

   def export(self):
      # logs that might be generated in the recipe folder at "export" time.
      # these would be associated with the recipe repo and original source of the recipe repo
      copy(self, "*.log", src=self.recipe_folder,
           dst=os.path.join(self.recipe_metadata_folder, "logs"))

   def source(self):
      # logs originated in the source() step, for example downloading files, patches or other stuff
      save(self, os.path.join(self.recipe_metadata_folder, "logs", "src.log"), "srclog!!")

   def build(self):
      # logs originated at build() step, the most common ones
      save(self, "mylogs.txt", "some logs!!!")
      copy(self, "mylogs.txt", src=self.build_folder,
           dst=os.path.join(self.package_metadata_folder, "logs"))

It does create the metadata folder in testing but when i use conan export-pkg . -of testing, it doesnot copy it to the cache.
Is this expected behavior or have I missed something?

[maitrey]

So as per the documentation conan build + conan export-pkg does not do it automatically. I need to run this command and add some copy commands after the package creation has finished:
conan cache path pkg/0.1:package_id --folder=metadata

[maitrey]

Package id could have multiple package revisions. How to get the accurate package id so that I upload the metadata to the write package id?

[memsharded]

So as per the documentation conan build + conan export-pkg does not do it automatically. I need to run this command and add some copy commands after the package creation has finished:
conan cache path pkg/0.1:package_id --folder=metadata

Yes, at the moment for export-pkg this is not copied automatically.
But quick question, if you already have all methods in place, "source()", "build()", "package()", why aren't you doing just conan create? The export-pkg is mostly intended for local developer testing (not for production) or for the cases when it is necessary to package some pre-compiled binaries provided by other means (not built by the Conan recipe itself).

Package id could have multiple package revisions. How to get the accurate package id so that I upload the metadata to the write package id?

If you did a conan export-pkg the package revision is the last one. the conan cache path by default will also resolve to the latest package revision. If that is not what you want, the conan export-pkg will output the package revision too, you can copy it, pass it to the conan cache path command, and it will give you the folder of that specific package revision too.

Note in any case, and this is very important, that in Conan 2.0, having more than 1 package revision for the same package_id is considered a failure in the process. Packages shouldn't have more than 1 package revision. If there is more than one, it means that something went wrong in the process or CI, and something that didn't have to be re-built was unnecessarily rebuilt. If anything changes in the source, the "recipe-revision" will be unique. If anything changes in the configuration, compiler, setup, etc, the package_id should be unique. Thus, for every recipe-revision+package_id, it is enough to build 1 package revision, no more.

[maitrey]

Because conan create does not offer -of which is needed for our developers. They need depedencies in the same location as the sources.
Here it says to use: https://docs.conan.io/2/devops/metadata.html package_id and not package_revision with conan cache.

[memsharded]

Because conan create does not offer -of which is needed for our developers. They need depedencies in the same location as the sources.

This would be a separate discussion. Why do they need the dependencies located with the sources? In the vast majority of cases this wouldn't be necessary, and using from the Conan cache is the recommended practice.

Also, the flows for development are typically different than the ones for package creation. Users can use conan install and conan build with -of, but still the recommendations:

• Developers shouldn't be the ones creating and uploading packages. Only CI should do that.
• Even if developers are creating the packages, the recommendation for production packages is to run conan create. This guarantees that the process is reproducible, but conan export-pkg doesn't.

You can do:

$ conan cache path pkg/0.1#recipe_revision:package_id#pkg_revision --folder=metadata

No prob with that. The code in the docs is showing the default of using the latest revisions.

[maitrey]

We use the same interface locally(conan build -of + conan export-pkg) as well as in CI/CD. Developers are not allowed to upload packages to the remote conan repo , that is done only by CI.

[memsharded]

We use the same interface locally(conan build -of + conan export-pkg) as well as in CI/CD. Developers are not allowed to upload packages to the remote conan repo , that is done only by CI.

Ok, sounds good that developers are not uploading.
Then yes, if CI is not doing conan create, it will need to copy the metadata, and you can do it if you want using the package-revision in conan cache path.

I think it might also be doable to implement the copy of the metadata in the export() and package() methods, knowing that it will be used in export-pkg. Both will have to the origin folders and the target folders, and knowing that the local folder are called "metadata", I think should be doable.

[maitrey]

Could you please give me an example what I should include in the export() and package() method to attach the meta data?
I also understand from the documentation why is save needed?

[memsharded]

save(self, "mylogs.txt", "some logs!!!")

This? This is needed just to emulate the generation of logs. It is not needed, in theory the logs will come from the build system.

The process for exporting the metadata could be like (not tested):

def export(self):
      # logs that might be generated in the recipe folder at "export" time.
      # these would be associated with the recipe repo and original source of the recipe repo
      copy(self, "*.log", src=os.path.join(self.recipe_folder, "metadata"),
              dst=os.path.join(self.recipe_metadata_folder, "logs"))

 def package(self):
    # logs originated at build() step, the most common ones
    copy(self, "*.logs", src=os.path.join(self.build_folder, "metadata"),
            dst=os.path.join(self.package_metadata_folder, "logs"))

[maitrey]

self.package_metadata_folder points to the output folder in the package method. And not to the package folder in the conan cache. Also all the test reports are located in the build folder and in the export method I donot know any variable that can access the self.build_folder.

[memsharded]

I have been double checking this. It seems that conan export-pkg is not defining the package_metadata_folder at this moment.
So if you want to implement this flow, you would need to do a copy to the folders defined by conan cache path.
We will have a look to try to improve it.

[memsharded]

Submitted #15126, that should fix, and would allow copying the package metadata in the package() method with conan export-pkg (test included)