Plugin system for compose

[GM-Alex]

What?

This pull request introduces an plugin system for docker compose. The plugin system uses python packages as plugin which
makes it easy to use the already existing classes and functions from compose.

Why?

There is always a missing functionality in your application software. A plugin system gives you the ability to extend
existing or add new functionalities. In addition it keeps the core clean, slim and result in more robust software.

How?

The pull request adds a new command plugin to docker-compose. The plugin has the following sub commands:

• install
  • installs a new plugin from a local file or an url (the file must be a tar or zip archive)
• uninstall
  • uninstall the plugin by the given id
• update
  • updates the plugin by the given id
• config
  • configures the plugin by the given id
• list
  • list all current installed plugins

Minimal plugin example

The following describes a minimal plugin setup and could be used as boilerplate code.

docker-compose.yml

There is a new entry for the docker-compose.yml called plugins where the config for the plugin can be set. If a
plugin is listed here it's required and it will be checked if the plugin is installed after the config was loaded. The
version option is optional and defines the lowest required version of the plugin. The setting made under options are
passed through the constructor of the plugin class as dictionary.

plugins:
  compose_plugin::
    version: 0.0.1
    options: 
      first_config_attribute: value
      seconde_config_attribute: value

Folder structure

• compose_plugin
  • init.py
  • plugin.json

init.py

from compose.plugin import Plugin

class ComposePlugin(Plugin):
    pass

plugin = ComposePlugin # Needed at __init__.py for every plugin

plugin.json

{
  "name": "Compose plugin",
  "description": "Compose plugin description.",
  "version": "0.0.1"
}

New decorators

The are two new decorator which makes it easy to monkey patch existing functions/methods (compose_patch) and add
new commands (compose_command).

@compose_patch(compose.service.Service, "build")

from compose.plugin import compose_patch

@compose_patch(object, "function")
def get_version_info(self, original_fnc, first_arg, second_arg):
    # do something
    return_value = original_fnc(self, first_arg, first_arg)
    # do something with return_value
    return return_value

compose_command

from compose.plugin import compose_command

@compose_command(standalone=True) # Flag to use the new command standalone or not (docker-compose.yml required).
def new_command(self, options):
    """
    New command

    Usage: new_command [options]

    Options:
        -o, --option    Command option

    """
    # Do the stuff for the new command
    pass

Restrictions

• Third party libraries needs to be stored inside of the plugin directory. That can be achieved with the following
  command executed at the plugin dir: pip install --target=${PWD}/libs THIRD_PARTY_PACKAGE
• Core packages which are not included at compose can't be loaded dynamically due the restrictions of pyinstaller

TODOs

• Provide a plugin repository which hold all plugins, something like docker hub for plugins
  • I think it would be enough to grep and hold all plugin.json files and use github for hosting the plugins
• Check if it's possible with pyinstaller to support also the dynamic loading of python core packages

Example plugin

For an example plugin see: https://github.com/GM-Alex/compose-bi

[GM-Alex]

@bfirsh @shin- @dnephin Because I haven't got some feedback until now, I just want to know if this PR is interesting for you guys. If yes I would keep it up to date.

[electrofelix]

Personally would to see this appear in docker-compose as it would allow those (like myself) who have some corner case projects that docker-compose almost covers but not quite fill, to implement a plugin to cover the project needs rather than having to wrap it with scripts or look at different tools to replace it completely.

[electrofelix]

Was http://bugs.python.org/issue12773 not fixed before python 3.3 released?

https://github.com/python/cpython/blob/3.3/Misc/NEWS#L4652 suggests it was, and testing against python 2.7.12, 3.3.5 and 3.5 locally there doesn't appear to be an issue in mutating the __doc__ string on user defined classes. So wondering if this part is really needed?

[electrofelix]

Should this be after setup_logging() in case you need to log something from the PluginManager class setup?

[andreyrusanov]

Just found you PR and curious: wouldn't it be more consistent to use yaml instead of JSON?

[GM-Alex]

@andreyrusanov I used json for the package description because other package managers like composer and npm using it. But there is no preference by my side and switching to yaml shouldn't be the problem. But first I need some feedback from the docker guys if the PR has a change to get merged, before I invest more time.

[bweston92]

I think this should be in your global gitignore file. Or edit $repo_dir/info/exclude

[shin-]

I understand this took time and effort to bring to life, and I thank you for sharing it with us and offering to contribute to the project. Unfortunately this is not something we can commit to supporting in the long term, and as such, I don't see us ever merging this.

Thank you nevertheless. Maybe this can live as a fork of the project if there's interest?

[electrofelix]

@shin- does that mean that you don't want to support hooks/plugins at all? see issues #374 & #1341, seems like people want a way to cover a small number of scenarios around the edges.

My personal use case would have been to use a different build command rather. Mainly around the possibility of optimization such as working out from a multiple set of Dockerfiles what base images are common to each and also what tag identifies the resulting container in case I want to use it as a base image for one of the other services as well. Additionally might want to template our Dockerfiles to resolve some information dynamically to be recorded as labels (e.g. have a git command use a specific branch and resolve to SHA1 that is stored in both the RUN command and labels for searching).

Maybe some enhancements might subsequently move into docker-compose but right now it seems like can't use docker-compose as the entry point, and it's unclear as to whether any changes would ever get accepted if we forked. Instead something else will have to do it, and that seems to make it more likely that adopting a different tool-chain entirely would be more useful in the long term.