[Discussion] Poetry to provide a decent hook-based plugin system

[webknjaz]

It's a follow-up for #672 (comment).

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

So setuptools have this nice hook points system where they allow replacing certain utils by some third-party implementations.
It can catch up with what's available by iterating through a list of entrypoints with names corresponding to hooks.
This is a fairly popular technique among setuptools-based dists. For example, pytest and tox support having third-party dists being automatically activated and used when installed given they "advertise" their entry points as required by the plugin system. These two tools use https://github.com/pytest-dev/pluggy to achieve this.

Back to Poetry, I think lots of people agree that providing ability to extend such a great tool is a widely wanted feature.
This would allow adding custom dist/package format generators, custom metadata generators etc. They could be used to retrieve long description from some esoteric formats or places converting that into supported markups. This would also make possible having a custom version constructor like popular plugins to setuptools facilitating version sourcing from Git/Hg (#140/#672). Or, another example, exporting a pinned env and direct dependencies into pip-friendly formatted requirements. And who knows, maybe someone will even try out having another resolver replacement?

Any thoughts on how this might look like?

[jgirardet]

Hi,
At the time I was searching a way to specify python version at install, I thought a bit about it.

What does we want ?

• add some custom command:
  the easiest I think. It just needs to look for new commands in the setup tools entry point and add it in console/application.py

• add some flags/option to existing command:
  already more tricky. should it be added for all commands ? I see something like it : a do_plugin_task() a start of handle which would apply some change before the builtin code. it isn't something hard except that poetry runs via cleo which define the option in the class docstring. so if you want to add an option to an existing command, it has to be changed at instantiation time. It means parsing the docstring change it and instantiate the "new" command. You need to have a clear logic in it since many plugins could change the same command.

What is different in poetry for doing this ?
Poetry is not installed via pip so using the standard way of entry point maybe not be ok (I'm not sure about it). It should come from a poetry command. someting like : poetry plugin install some_plugin and the plugin could be in pypi.

Should we try to implement this ?
I would be ok, maybe others too but my feeling is that @sdispater since the beginning of the project, likes to implement himself the new core features of poetry. It's me feeling, I'm not sure about sebastien's mind about it.

Anyway, the idea of a plugin system is itself a great thing !!!

[sdispater]

Just for information: I am currently rewriting the core of Cleo to make it more flexible and pluggable.

Regarding the actual plugin system, I am not sure what it will look like just yet. But the general idea is to be able to add commands and command options to Poetry like @jgirardet said. I also think being able to hook into the Poetry object creation would be good since it would allow things like #672.

Now, i will likely implement the plugin system myself and make a PR to get feedback.

This is something I had in my mind from the start but there were more pressing matters than this but since Poetry is on the path of stabilization I think it's a good time to start experimenting with it.

I can't guarantee that it will make it to the 1.0 release but I will try.

[webknjaz]

Cool, thanks for the info.

[webknjaz]

Hey @sdispater, is https://poetry.eustace.io/docs/plugins/ something that implements what you wanted? Is it already possible to hook into that?

[webknjaz]

@sdispater any updates?

[funkyfuture]

while @webknjaz inital post points to uses that would extend poetry's functionality as a packaging tool and kinda venv manager, the idea to allow custom commands points to a different path. from the discussions that i came across, they all aimed at either extending poetry to a task runner or to interact with version information stored in the config file. for these cases, poetry would only act as a framework to implement cli commands around a pyproject.toml. at that point poetry would be two different things. in a better, possible world poetry and these other command implementations would share a common framework.

[vBLFTePebWNi6c]

Hi, @sdispater ! Our team is looking for pipenv's alternatives and poetry seems great, if it'd have plugin system, which is mandatory for our development process. I've seen that related PR is in progress -- can you give us eta for this feature?

[mrh1997]

Another use case for hooks would be calling preprocessors like cython or template precompilers.
Ideally these libraries could provide a plugin, that is called automaticially before the "build" command. The plugins could retrieve the details which files to process and how from a custom section of the pyproject.toml.

One important thing is that this requires the plugin interface living within the projects python virtual environment, while the plugin interface discussed above lives in the python interpreter where poetry is installed.

The big advantage of a plugin interface living in the project interpreter would be, that it not only solves the preprocessing-requirement, but also the other requirements of this issue on a per project basis. This would mean that adding a plugin is as simple as adding a development dependency to pyproject.toml. It will ensure that all developers working on the project are using the same plugin and plugin-version.

[jedie]

Is there a pre publish hook or any work a round or hack to get this?

EDIT: i created https://pypi.org/project/poetry-publish/ and used poetry via subprocess...

[gsemet]

That would be really useful if plugins could be installed inside the virtualenv of the package, not system/user wide!

[PetterS]

Would be nice if the plugin system allowed customizing the execution of package installation. This would allow a significant speed-up by enabling parallel installation: see https://github.com/python-poetry/poetry/pull/2374/files

[sdispater]

@PetterS Improving the installation time (and a refactoring of the installation logic) is planned and already part of the roadmap for the 1.1 release (see #1856).

[PetterS]

Cool, did not realize this was covered in "Download distributions separately before installing them.".

[fredrikaverpil]

Is it possible to somehow hook into poetry today, which e.g. makes it possible for me to inject the version from setuptools_scm into pyproject.toml just before building a wheel using poetry build?
Preferably, also some way of undoing the change to pyproject.toml after the wheel was built.

I'm looking for a "quick fix" until there is a proper setuptools_scm support with poetry.

This is what I do currently, but it's not at all very nice...:

# scm_version_hack.py

from setuptools_scm import get_version


def write_toml(filepath="pyproject.toml", reset=True):
    contents = []
    new_contents = []
    with open(filepath, "r") as infile:
        contents = infile.readlines()
    for line in contents:
        if line.startswith("version ="):
            if reset:
                new_contents.append(f'version = "0.0.0"  # DO NOT CHANGE VERSION HERE\n')
            else:
                new_contents.append(f'version = "{get_version()}"\n')
        else:
            new_contents.append(line)
    with open(filepath, "w") as outfile:
        outfile.writelines(new_contents)


def main():
    print(get_version())


if __name__ == "__main__":
    main()

python -c "import scm_version_hack; scm_version_hack.write_toml(reset=False)"
poetry build
python -c "import scm_version_hack; scm_version_hack.write_toml(reset=True)"

[mtkennerly]

@fredrikaverpil, you may be interested in this pseudo-plugin I made: https://github.com/mtkennerly/poetry-dynamic-versioning

[fredrikaverpil]

Thanks @mtkennerly that looks nice! 👍
However, we're a number of people who then need to make sure we have the same local setup (as well as the CI), and I guess this will break if poetry is updated?

So I think then my current workaround is probably better in the interim.

[gsemet]

I wonder why this feature would not be provided by Poetry out-of-the-box, of course on a voluntary basis.

[Conchylicultor]

Is there any update on this ? We deploy our project in two pip packages (a project-nightly released every day and project containing the stable version). There is currently no way to support this in poetry, which is a blocker for us.

@abn , it seems @sdispater isn't contributing anymore. Will someone take over the issue ?

[miigotu]

Example (simple but useful initial concept) project utilizing the new plugin system is at https://pypi.org/project/poetry-date-version-plugin/ and https://github.com/miigotu/poetry-date-version-plugin

#3733 makes the possibilities endless

It requires installing poetry from master pip install git+https://github.com/python-poetry/poetry in order to work until next poetry release.

[gsemet]

Does this means the plug-in can be located inside the virtual env and not on the same installation than poetry ?

[miigotu]

Does this means the plug-in can be located inside the virtual env and not on the same installation than poetry ?

Yes, it can be installed next to poetry itself with pip even

[gsemet]

What I would expect is not to pollute the system install and everything works from the user’s virtualenv

[miigotu]

What I would expect is not to pollute the system install and everything works from the user’s virtualenv

It works either way. If poetry is inside a venv, so are the plugins.

[bkhl]

What I would expect is not to pollute the system install and everything works from the user’s virtualenv

I was hoping as well that it would be possible to enable a plugins specific to a project.

However, it seems difficult to allow them to run within the virtualenv that you created with Poetry, since that might be running an entirely different Python version than the one that Poetry is running in.

[miigotu]

That's when you run poetry plugin add have you tried the feature?

[takeda]

@miigotu, @gsemet meant whether you can include a plugin in your project dependencies that will be invoked within your project's virtualenv. Kind of like with setuptools, you could include setuptools_scm in setup_install to get project version from git.

Anyway, I looked at the PR and the answer unfortunately is that plugin needs to be installed in the same environment from which poetry runs. Then your pyproject.toml needs to declared it wants to use it. So ultimately is similar what existing unofficial plugins were doing, just that now there's an official way to do it. This kind of sucks for reproducibility (another developer will need to also install this plugin), but it is still something.

I'm hoping that maybe in the future when you run poetry install from your project, poetry would display something like:

This project requires 'super-duper-plugin >= 1.0'. Install? (y/N)

[gsemet]

If you want this plugin system to work you need to treat this plugin like a “dev-package”, ie, just install it within the virtualenv.
Installing it in the same place than poetry is pretty much useless.

My use case is auto versioning. I just want this feature for the moment it needs to be installed on the user environnement (poetry-dynamic-versioning). So I have a ugly bootstrap script that installs it with a fixed version, like the old days where no lock files existed...

the plugin dependency should be declared and controlled in the lock file, so that two different project could use two different versions of the same plugin.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.