Plugins v1

[regisb]

This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.

Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.

This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.

The plugin v0 API remains supported for backward compatibility of
existing plugins.

FAQ

Is this PR ready for comments?

Yes! If you are a plugin developer, we would like to hear what you have to say.

Why is this PR so large?

Because I refactored the entire code base to make use of the v1 API. Tutor core now uses its own extension mechanism, in the spirit of dogfooding. We do that because the eventual long-term goal is to extract the extension API from Tutor in a different package, such that it can be used to deploy not just Open edX, but other software as well. (see this issue for more information)

Where should I start?

I'm glad you're interested :)

1. First, please have a look at the new plugin development tutorial. Basically, read all changes in the docs/ folder.
2. Have a look at the new hooks API, in the tutor/hooks/ folder.
3. Have a look at the changes made in the commands module: all files in the tutor/commands/folder

I think that at this point you should already have plenty of questions and comments...

When should this be merged?

All official plugins should be migrated to v1 prior before the Nutmeg release (June 9th 2022), so April 30th would be a good deadline.

Will this PR break my plugin?

No. Existing plugins should keep working as expected. If they don't, then it's a bug. Please report them in the forum, as usual :)

How long will the v0 API be supported?

A long time. We want to leave time to all developers to migrate their plugins from v0 to v1. I don't anticipate that this will cause any issue, because under the hood the v0 plugin API already uses the v1 API.

What can I do with v1 that I couldn't with v0?

Lots of stuff! This PR does not introduce many new features by itself, but I expect that in the future it will become very easy to make profound changes to how Tutor works using just plugins. Here's a sneak peek of what I have in mind:

• Open edX themes as plugins
• Plugin-extensible argument parsing, for instance to conveniently automount host folders.
• Extensible tutor local and tutor k8s CLI.
• Plugin debugging and troubleshooting, via another plugin naturellement.

Done

• Do not load commands from plugins which are not enabled.
• Load enabled plugins once on start.
• Implement contexts for actions and filters, which allow us to keep track of
  the source of every hook.
• Migrate patches
• Migrate commands
• Migrate plugin detection
• Migrate templates_root
• Migrate config
• Migrate template environment globals and filters
• Migrate hooks to tasks
• Generate hook documentation
• Generate patch reference documentation

TODO

• Amend the API such that we can write tutor.hooks.Filters.MY_FILTER.apply(...)
• Maybe rename Action.on() to Action.add()?
• Make it clearer in the docs that actions and filters are different types of hooks.
• Clarify in the docs that contexts is very much an internal concept that most plugin developers don't have to care about.
• Don't talk about enable/disable in the code, but pick different terms.
• Decide whether to keep action/filter *args. EDIT: we should keep them.

[regisb]

This is ready for review @overhangio/tutor-developers. I understand this is a huge PR, and it requires some sort of explanation. We are currently planning a meeting with @kdmccormick so that everyone understands what's at stake here, and what is the technical solution that I've ended up implementing. We'll decide on a meeting date and then plugin developers will be invited to the meeting. Meanwhile, you can learn more about that project by having a look at this comment: openedx-unsupported/wg-developer-experience#32 (comment)

[regisb]

If other plugin developers are interested in learning about the new plugin API, we'll be discussing it with Kyle & Rebecca today March 29th at 12:30 UTC: https://us02web.zoom.us/j/85637351768?pwd=Y1htdFlLNnRTZzd6VXQwNW45eTArUT09
FYI @kdmccormick @rgraber @bradenmacdonald @fghaas @BbrSofiane

[fghaas]

Thanks for the heads-up @regis! I'll try to make that but it comes at slightly short notice for me so I'm not sure if I'll be able to, hence I'm putting a few thoughts down here in writing. I hope that's OK.

In brief, I admire the rationale behind this, and I think it's a good approach. But, and this is completely separate from a discussion on the technical merits, to be honest I am a bit skeptical about the timing.

Consider that since the December Open edX Maple release which made Tutor the default mode of deploying Open edX, a bunch of people would have been scrambling to achieve feature parity with edx-configuration, so that they can move their production platforms to Tutor. Since Tutor has always followed the policy of keeping its basic feature set simple, and exposing other features with plugins, that means that people have been busy writing and testing plugins since then (we've written 7 so far). Are you sure it's a good idea to introduce a new plugin API just now, less than 4 months after the Maple release, essentially forcing people to start updating/rewriting them? (Yes I understand you want to keep backward compatibility with the v0 API. But you wouldn't want to do that for long, see xkcd927 🙂).

My point is just that after several years in limbo — with edx-configuration being "quietly" deprecated long before it officially was, no endorsed alternative coming out of edX, and there having been being months of back-and-forth until Tutor was finally adopted as the new community development platform — community Open edX deployers could be forgiven to desire some degree of stability in the deployment tools. So, is the v0 plugin API perhaps "good enough" for a few more months until people have completed their Maple/Tutor transitions, for a new plugin API to be introduced thereafter?

[regisb]

I understand your concern Florian and I agree that the timing is not great. Several things are happening at once with Tutor: system administrators are learning to switch from the native installation and edX/2U is moving away from the Devstack. The latter project is straining the v0 plugin API beyond its capabilities, as we can see from these issues that were collected by Kyle: https://github.com/overhangio/2u-tutor-adoption/issues/

The goal of this new v1 API is to make it easier for developers to use Tutor and to customise their platforms with plugins. It would be counter-productive if it actually made people's life more more difficult.

My understanding is that to make this transition successful we need to work on three key aspects:

1. API stability (including the v0 API, for which we don't expect to set a deprecation date)
2. v1 features that are actually useful and productive
3. communication and pedagogy to help people transition from v0 to v1

We can cover items 1 and 2 by preserving support for the v0 API and resolving issues listed in the 2u-tutor-adoption project. To address item 3 we are planning to:

• organise a tutor dev quickstart tutorial at the conference: https://openedx2022.sched.com/event/zx6P/tutor-dev-quickstart I'm hoping to re-use the material in later live coding sessions (see below)
• improve the Tutor documentation
• organise live coding/tutorial streams on Twitch/Youtube: this is something that I've played with in the past and I'm planning to do it more consistently now that I finally have an Internet connection that doesn't suck (yay fiber!)

I'll flip the question: you and your team have made several significant contributions to Tutor and I'd like you to keep doing that. How can I make your life easier and more productive?

I'll try to make that but it comes at slightly short notice for me so I'm not sure if I'll be able to

It's OK if you can't come; this is going to be a preliminary meeting, and I expect that there will be at least another one with Tutor maintainers before we merge this PR.

[regisb]

Avoid overloading "enable" and "disable". I think enable/disable should just mean adding or removing a plugin from the list in config.yml. To describe the process of adding or removing a plugin's callbacks, could we invent new terms, for example "activate" and "deactivate"?

@kdmccormick what do you think about "load/unload"? Or is that too generic?

[kdmccormick]

@regisb I like load/unload 👍🏻

[regisb]

@kdmccormick I've made the requested changes, please let me know what you think. I've tried to improve the wording across the docs and the codebase. We can always improve it later, but we should be very careful with the hook constant names, which are not supposed to change with time (although I guess that we can take some liberties).

Regarding the suppression of hook *args: at first I agreed with you, but then when I attempted to get rid of them I realized something: if we make use only of keyword arguments, then we need to provide a default value for each one of them. And then, every hook needs to check whether the keyword argument is correctly defined or not. This adds too much complexity to my taste, so I decided to keep *args. We can discuss this further if you like.

[kdmccormick]

I think the plugin guide is awesome. I love that it guides a user from basic concepts all the way to adding a new service. I have suggestions for it, but like you said, we can improve it after this PR lands if you'd rather not address all my feedback on it right away.

I agree that the hook names are the most important thing to hammer out right now. I have suggested some changes to the action names, and I'm chewing on the app:task:* filter names currently.

if we make use only of keyword arguments, then we need to provide a default value for each one of them.

Python 3 has a funky syntax to solve exactly this problem, if you were so inclined:

>>> def f(*, x, y):
...     return x + y
... 
>>> f(1,2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() takes 0 positional arguments but 2 were given
>>> f(x=1,y=2)
3
>>> f(x=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() missing 1 required keyword-only argument: 'y'

[MaferMazu]

I'm starting to see this PR and at the moment I have the following questions:

• Will yaml file plugins still exist?
• I understand that there will also be some "simple" plugins (not packages) that will be .py files and with them you will also be able to add patches as mentioned in tutorials/plugin, right?
• Will the structure of the plugins that are python packages change a lot?
• Is there a public example of a v1 plugin package?

[regisb]

To answer your questions @MaferMazu:

Will yaml file plugins still exist?

It will not be possible to create YAML plugins with the V1 API, but existing v0 plugins will remain supported for a long time.

I understand that there will also be some "simple" plugins (not packages) that will be .py files and with them you will also be able to add patches as mentioned in tutorials/plugin, right?

Yes, exactly. This will be the new "lightweight" way of creating simple plugins, similar to YAML plugins, but not subject to the same limitations (this one for example).

Will the structure of the plugins that are python packages change a lot?

Not drastically, no. The plugin.py will be different, as well as the setup.py entrypoint, but I expect that the differences will be relatively straightforward to resolve.

Is there a public example of a v1 plugin package?

Not yet. I'm working on upgrading the plugin cookiecutter to generate valid v1 plugins (see this issue).

Regarding your comment @kdmccormick:

Python 3 has a funky syntax to solve exactly this problem, if you were so inclined:

That's a bit too funky for me :) I had no idea that this syntax existed!

[regisb]

I think I've exhausted the list of things to do here. I'll merge now 🏴‍☠️

[kdmccormick]

Woohoo! 🎉