[question] Recommended way of calling conan from python program

[simon-rechermann]

What is your question?

Hi,
I want to write a conan wrapper around the conan CLI for conan v1 and conan v2.
I see two possible solutions:

1. Using the subprocess module of python to directly call the conan cli. So something like this:

# Example: conan create command
subprocess.Popen(["conan", "create", ".", "version@user/channel"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # conan v1
subprocess.Popen(["conan", "create", ".", "--version version", "--user user", "--channel channel"], stdout=subprocess.PIPE, stderr=subprocess.PIPE) # conan v2

2. Accessing the conan api. So something like this:

# conan v1
from conans.client.api import Conan
.....

# conan v2
from conan.api.conan_api import ConanAPI
......

I'm not sure what the better approach is. Regarding approach 2 I'm not sure if the ConanAPI is still WIP or if it's even meant to be used like this? I haven't had a deeper look into the conan v1 API but regarding conan v2 I saw that for example there is no high level CreateAPI and the create CLI command uses more fine granular API commands from other APIs(ExportAPI, GraphAPI, InstallAPI, ...) for package creation. This would make it easier to go for approach 1 as it seems like the CLI provides a simpler/more high level abstraction than the API.
Is there a general recommendation or programs that try to do something similar?

Thank's in advance for your help

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @simon-rechermann

Thanks for your question.

It depends on what you need.

For most cases, the recommended way would be the Conan PythonAPI that has been made public and documented in Conan 2: https://docs.conan.io/2/reference/extensions/python_api.html. The documentation is still WIP and it might still suffer some changes as a result of the experimental status, but we try not to break if possible.

Also, for most cases, it is further recommended to not use the PythonAPI "bare", but use the new custom commands extensions, that allows you to create and manage your own commands built on top of the PythonAPI and other commands. Check: https://docs.conan.io/2/reference/extensions/custom_commands.html. This is a very powerful approach, as it also allows to publish, share and update your custom commands with the conan config install or conan config install-pkg very easily.

Regarding the usage of other commands, the Conan 2 PythonAPI also provides the "CommandAPI" in https://docs.conan.io/2/reference/extensions/python_api/CommandAPI.html, which is specifically intended to call other commands, and interface at the Python level, which will be way more convenient than a subprocess approach, something like:

            @conan_command(group="custom commands", formatters={"json": format_graph_json})
            def mycommand(conan_api, parser, *args, **kwargs):
                \""" mycommand help \"""
                result = conan_api.command.run(["create", ".", "--version=1.0.0"])
                return result

This is a very new addition, and docs are not there yet, but feel free to try it and ask questions as necessary.

[simon-rechermann]

Hi @memsharded

Thanks for your answer.
I still don't really understand how custom commands or the CommandAPI can help me achieve my goal.
The example you showed with the @conan_command creates a new conan cli command and the mycommand function is not meant to be called from python code or do I get this wrong?
I tried the following:

from conan.api.conan_api import ConanAPI
conan_api = ConanAPI()
conan_api.command.run(["remote","list"])

and I get this error: 'NoneType' object has no attribute '_commands'
for this line:
commands = getattr(self.cli, "_commands") # to no make it public to users of Cli class
of the CommandAPI.
So it looks like the CommandAPI need a self.cli to work properly?
How do I have to init the CommandAPI to be able to use it?

[simon-rechermann]

I see using the CLI works but I'm not sure how to do this with the CommandAPI?

from conan.api.conan_api import ConanAPI
conan_api = ConanAPI()
conan_cli = Cli(conan_api)
conan_cli.run(["remote","list"])

Looks a bit strange to me that the CommandAPI is depending on the CLI as I thought the CLI is a layer built on top of the ConanAPI

[memsharded]

The example you showed with the @conan_command creates a new conan cli command and the mycommand function is not meant to be called from python code or do I get this wrong?

The custom commands are already Python code. The idea is that you don't need to have some Python scripts with a custom interface, but you can structure and manage your Python scripts as Conan commands. Instead of launching your python script as python myscript.py .... you do conan mycommand and benefit from:

• Standard argument parsing aligned with other Conan commands
• The commands are listed in conan -h
• Same --format output management
• Automatic injection of the api, no need to instantiate anything
• Manage the commands with conan config install/install-pkg together with the rest of the Conan configuration (remotes, settings, profiles, etc)

and I get this error: 'NoneType' object has no attribute '_commands'

yes, the CommandAPI needs an entry point to the Cli object, because that is the one that contains the defined commands. Something like:

conan_api.command.cli = conan_cli

Looks a bit strange to me that the CommandAPI is depending on the CLI as I thought the CLI is a layer built on top of the ConanAPI

The thing is that the commands are modular and extensible, you can add your own commands. Then, there is no hardcoded definition of commands, and it is the Cli class the one that loads them and store them, so in order to call commands, it is necessary a Cli object, as it needs a back reference to the list of commands to be able to call other commands.

In any case, if you use the custom commands, all of this is automatically managed. So far, users that have gone for the custom commands are very happy about it, it is very recommended, better than just python scripts using the api.

[simon-rechermann]

Thanks for your help! :)
I'm sure the custom commands are very useful for a lot of use cases. However, I would like to have a python wrapper around the API that can for instance be shipped as a python wheel package and from there be used by other python packages. So not just a small python or bash script. I think for this specific use case I'll end up at the subprocess approach or using the CLI object's run method, even when calling custom commands from there.
Or I use the CommandAPI but even with what you proposed conan_api.command.cli = conan_cli it does not work for me. Am I missing something here?

Python 3.10.14 (main, Apr  6 2024, 18:45:05) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from conan.api.conan_api import ConanAPI
>>> from conan.cli.cli import Cli
>>> api = ConanAPI()
>>> cli = Cli(api)
>>> api.cli = cli
>>> api.command.run(["remote","list"])
Traceback (most recent call last):
  File "/home/recherma/dev/mrt.rfu.logging.interface/venv2/lib/python3.10/site-packages/conan/api/subapi/command.py", line 20, in run
    command = commands[current_cmd]
KeyError: 'remote'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/recherma/dev/mrt.rfu.logging.interface/venv2/lib/python3.10/site-packages/conan/api/subapi/command.py", line 22, in run
    raise ConanException(f"Command {current_cmd} does not exist")
conans.errors.ConanException: Command remote does not exist

[memsharded]

I'm sure the custom commands are very useful for a lot of use cases. However, I would like to have a python wrapper around the API that can for instance be shipped as a python wheel package and from there be used by other python packages. So not just a small python or bash script. I think for this specific use case I'll end up at the subprocess approach or using the CLI object's run method, even when calling custom commands from there.

Sounds good, thanks for the explanations!

As this is not documented yet, let me check it, try it, and I will probably propose a PR to the docs with the necessary steps.

[memsharded]

This is the working implementation atm:

from conan.api.conan_api import ConanAPI
from conan.cli.cli import Cli


api = ConanAPI()
cli = Cli(api)
cli._add_commands()
result = api.command.run(["remote", "list"])
print(result)

The missing piece was a _add_command call.
It happens that the custom commands were so succesfull, we didn't have to make this public yet.

I will do a PR to clean the interface a little bit. You might use this solution in the meantime, but please take into account that it will break next release.

UPDATE: The line api.command.cli = cli was unnecessary, I removed it.

[memsharded]

#16213 will make add_commands public, it will also do a PR to the docs to add this, and it will close this ticket when merged, it will be in next Conan 2.4. Thanks very much for your feedback!