[feature] Add a deploy() method to conanfile for application specific deployment logic

[samuel-emrys]

What is your suggestion?

Hi Team,

Bottom Line Up Front

The proposal

Add a deploy() method to the conanfile, that would allow the author to specify very specific deployment instructions for applications where a more general deployment script is not appropriate. This might be invoked as follows:

$ conan install --requires="myapp/1.0.0" --deploy /path/to/deployment/location

This does not necessarily need to conflict with --deployer (both may be useful for different applications, i.e. a custom deployer to aggregate licenses or generate a SBOM is not mutually exclusive with deployment for production usage).

The use case

It would be good to have a method of writing a deployment mechanism for a single application. My specific use case here is for a GUI application that is unlikely to be invoked via the command line or with a virtual environment activated. Without knowing a heap about these applications, it might be useful to think of how an application such as Adobe Photoshop or Blender might be managed and deployed using conan.

For this case, these applications have the following properties:

• Highly specific project structure that needs to aggregate dependencies and store/read configuration from the deployed location
• Not useful as part of any developer process - this is an end-point deployment
• Unlikely to be executed from the command line
• Unlikely to use virtual environments (even though these are useful, some legacy applications won't play nicely)

The current method of doing this might be to use option (3) below, however it seems to me that this is at the wrong level of abstraction. If I have a number of applications like this, each would require their own deployer to live in the cache, which would necessitate distribution of every applications deployment recipe with global configuration. There is a UX issue here that could lead to one of these highly specific custom deployers being used to deploy packages they're not intended for, and would fail and throw a bunch of errors. It seems to me that for highly application specific deployment requirements, this deploy script/recipe is best placed with the conanfile of the applications being deployed itself.

Context - A summary of alternatives

Currently, there are a few of methods that we have available for application deployment:

1. Don't deploy

For many use cases, especially when using build tools or other command line utilities, this may be sufficient:

conan install --tool_requires="cmake/[>3.25]"
conanbuild.bat
cmake.exe --version
# when you are done
deactivate_conanbuild.bat

And this works great when the application can be engaged from the command line. This does not satisfy the use case of system level deployment of a GUI application that expects to be bundled with its dependencies somewhere.

2. Built in deployers

Otherwise, for deployment, the build in deployers are great for simple applications or applications that adhere more to the unix philosophy in terms of application structure

$ conan install --requires="cmake/[>3.25]" --deployer=direct_deploy
$ tree direct_deploy -L 3
direct_deploy
└── cmake
    ├── bin
    │   ├── ccmake
    │   ├── cmake
    │   ├── cmake-gui
    │   ├── cpack
    │   └── ctest
    ├── conanbuildenv-x86_64.sh
    ├── conanbuild.sh
    ├── conaninfo.txt
    ├── conanmanifest.txt
    ├── conanrunenv-x86_64.sh
    ├── conanrun.sh
    ├── deactivate_conanbuild.sh
    ├── deactivate_conanrun.sh
    ├── licenses
    │   └── Copyright.txt
    └── share
        ├── aclocal
        ├── applications
        ├── bash-completion
        ├── cmake-3.25
        ├── emacs
        ├── icons
        ├── mime
        └── vim

You can invoke the generated environment files and use them temporarily, or you can add them to PATH in a more permanent way if desired. This may work excellently on unix like systems where these structures can be injected into the system easily.

However, it's not flexible enough to satisfy the use case of system level deployment of a GUI application that can't be easily invoked by command line, or where the deployment is targeting users that are not comfortable with the command line.

3. Custom global deployer

These are also great when built in deployers don't do what we need them to, and shine especially in situations where the deployer can perform a deployment that applies to a large number of packages. I.e., if this deploys cmake in a way that would be equally as valid as deploying meson or protobuf.

$ conan install --requires="cmake/[>3.25]" --deployer=mycustomdeployer
$ conan install --requires="meson/[>1.0.0]" --deployer=mycustomdeployer # Great! adheres to DRY principle and we get re-use of common deployment logic

This does satisfy the use case - I can write a highly specific custom deployer for my application here. The objection I have to this method is that it lives within the $CONAN_HOME/extensions/deploy directory and can be applied to other applications where it would not make sense. To illustrate, I could:

$ conan install --requires="myapp/[>3.25]" --deployer=myapp # This would work great

Which would work, but I could also:

$ conan install --requires="myotherapp/[>3.25]" --deployer=myapp # Error! This makes no sense

If I need to write a deployer that is highly specific to a single application, I think that there should be a way of coupling this with the package recipe rather than having to distribute multiple components (the deployer and the recipe) separately to make this possible.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[stackfun]

This would be useful for our use cases too.

Currently when we create a product installer, it pulls down Conan dependencies, copies files while changing the layout slightly, add a few installation scripts, then create/publish a mega-package. It's a pretty big waste of storage and network throughput.

@memsharded mentioned that what we could have done instead, is to create a conanfile.py with a requirements, generate, and build function, publish only the recipe, then rely on conan install --build=missing . While this gets us mostly there, I wonder if we would be abusing some Conan features, especially since we want to call some installation scripts. @samuel-emrys's proposal seems like a better fit.

We were also exploring using Conan as a way to distribute and install common tools to developers which may not be related to the C++ build. We'd prefer developers to stop installing random tools from the public internet, and only the ones we have vetted. The deploy() function seems useful here too.

Note that we only use our private conan server and don't rely on cci at all.

[samuel-emrys]

Hi @memsharded, is this something that could be considered?

[memsharded]

Hi @samuel-emrys

Yes, sorry for the delay.
Sure, this is something that could be considered. I can't guarantee anything, but I'll take it with the team and discuss about it. Thanks for the suggestion!

[memsharded]

This would be very early stage, but if you want to give it a try for early feedback, very welcome: #15172

Use --deployer-pkg=* to specify the packages that we want to deploy. I think it is a valid use case that there could be more packages with deployment instructions, and having a explicit argument to specify which one or which ones have to be deployed makes sense.

[memsharded]

#15172 has been merged for next 2.0.15, feedback welcome!

[stackfun]

Awesome, thanks so much!

We couldn't wait, so we implemented a custom Conan command, but I think we can completely switch over to Conan's implementation if we could pass custom arguments to the deploy function. We use click to define our deploy parameters.

    @click.command()
    @click.option("--foo", required=True)
    def deploy(self, foo: str):
        copy(self, "*", src=self.package_folder, dst=self.deploy_folder)
        self.run(f"{sys.executable} {self.package_folder}/scripts/install.py --foo {foo}")

Our internal command is invoked like conan <org_name>:deploy -- --foo=bar (there's a default value for deploy folder in our implementation)

Since not everyone uses click, maybe custom params can be passed in as args to the deploy function?

    def deploy(self, *args: str):
        # parse command line args here

We also needed to invoke an uninstall script, so we have another hook in the conanfile that we invoke for that.

[memsharded]

You can pass custom arguments to any Conan recipe method, with full control to pass different arguments to different recipes too. The mechanism is user.xxxx confs:

Use $ conan install ... --deployer-packages=mypkg* -c user.myteam.deploy:mydeployconf=myvalue

Then

def deploy(self):
     myvalue = self.conf.get("user.myteam.deploy:mydeployconf")
     ...

The injection of command line arguments to recipe methods would break the design and interfaces, I am afraid that approach won't be possible, but the conf approach should achieve the same, with some extra goodies (parameter type validation, appending, prepending of values, aggregation of values from multiple profiles, etc).

[stackfun]

--conf approach should work for us!

[samuel-emrys]

Thanks for all the work on this @memsharded - this is looking great and looks like it will suit my needs (I'll find out when I test it in anger for a large application, but it works great for something trivial). I've left a couple of review comments in #15172 (comment) for your consideration.