Plugin system for obscure/non-universal features

[ethantkoenig]

I propose adding a plugin system to gitea, which would allow gitea users and developers to add new features while keeping the core gitea codebase small and maintainable. Please read my proposal, and let me know what you think.

Motivation

As gitea grows, there will be requests for more and more features. Many features will be relatively peripheral; while necessary/useful to some users, they will not be used by other users. An example of such a feature is #2211. I'm afraid that gitea will eventually become a bloated collection of incoherent features. Even if these features can be enabled/disabled in a config file, they still make the codebase more complicated and harder to maintain.

Meanwhile, many users of gitea may want to perform small-scale changes and customization to their personal forks. These could range from logo changes to small features that are for their specific use-case (and thus wouldn't make sense in upstream gitea). It would be nice if such users could make these changes without facing merge conflicts when they pull from upstream.

I believe plugins could solve both of these problems.

Specifics (I'm open to suggestions and discussion)

Plugins would be in-process (included during compilation). Plugins would interact with "core" gitea in the following ways:

• Plugins can update templates (this would happen during start-up). This allows plugins to add new UI elements
• Plugins can register new routes
• Plugins can install listeners/callbacks that are triggered when various events happened (e.g. new content is pushed to a repo, which could be useful to a plugin for full-text repo search)
• Plugins can add new tables to the database. Unfornuately, there doesn't seem to be a clean way for plugins to alter existing tables, since they corresponding struct in "core" gitea would need to be updated.

Unanswered Questions

1. What if two plugins rely on different version of the same dependency?

I don't have an answer right now, but hope to investigate possible solutions soon. The answer also depends on whether we end up switching to golang/dep (#2218)

[sapk]

We could use the plugin method introduce in Go1.8 : https://golang.org/pkg/plugin/

[sapk]

Futhermore, I propose :

• Plugins can add command line command (like: import github/bitbucket/gogs)

This would permit to add script from community wihout bloating main prog.

[lafriks]

@sapk Go plugins are supported only on Linux

[sapk]

@lafriks I have to correct what I have said on compiled plugin and vendored deps : golang/go#20481

This should be fix but not yet ready.

[sapk]

Windows and macos plugin build are an objectif but not yet planned and no one on it. 😞
I would still suggest it and explain that it is only possible for linux host in the wait of golang improvement.
Maybe there are better solutions ?

[andreynering]

Gogs cross-reference: gogs/gogs#2438

First, I completely agree the we should not bloat Gitea with more and more small features.

About the plugin system, it's also hard to implement. Go has interpreters for other languages we could look:

• JavaScript: https://github.com/robertkrimen/otto
• Lua: https://github.com/yuin/gopher-lua
• Lua: https://github.com/Shopify/go-lua
• Bash: https://github.com/mvdan/sh (this one may not be suitable for the plugin system, but for allowing running bash commands that will run on Windows, too)

We should look for an approach that is either easy to maintain in the Gitea core and also for plugin developers.

[sapk]

https://github.com/hashicorp/go-plugin suggested on gogs issue is also a good solution. It support some few good feature like negotiation for compatibility between main and plugins (and is planning "plugin fetching"). And plugin could be develop in various languages just need gRPC interface.

[sapk]

gRPC permit support for plugin triggerring event and not wait to be called.

[jonasfranz]

I think that we only need a better filtering of features and we have to know what we want to implement and what not. But I disagree with you statement that #2211 is only used be a "small minority of users" because it makes gitea interesting for small businesses that need such features. And my apprehension is that the features will not be such good integrated or have a good quality if they come via plugin.
I agree that there should be a simple possibility to add small additions like #2221 but I do not think that it would be a good option for (in relation) big features like #2211.

[lunny]

I agree with @JonasFranzDEV enhancement of issues will let small company remove their redmine and such others.

[ethantkoenig]

@JonasFranzDEV @lunny Apologies if I came across as unfairly criticizing the timetracking feature. I didn't mean to say that the feature is useless, but only that it is peripheral to the main purpose of gitea, and that large subset of gitea users do not need it. I've updated the wording in my description accordingly.

I'm not sure "we just need to do a better job deciding what to implement" is a viable solution. Who decides what is worth implementing? What if user X needs feature Y, but it is decided that feature Y will not be implemented? User X's only option is to implement feature Y on his/her own fork, which is problematic as I described in the description.

Finally, I disagree with the "too big for a plugin" argument. The fact that the timetracking feature is large means it will add even more overhead and complexity to the codebase, which I believe further underscores the need for moving it to a plugin.

[lafriks]

@ethantkoenig plugins are good thing to have but also come with additional problems like it is harder to change core functionality/design/structure as such changes can and most probably will break plugins and is harder to test

[sapk]

@lafriks It also give us room for change because with too much function integrate in core we lock the structure of code and giving a standard interface (with versioning) for plugin will allow function to evolve at there own rythm. If a user want to keep a func implemented by a plugin that does not support the last version of core, it become the choice of the user to stay with an older version in the wait of an update of the plugin (or contribute to the update).

I think we all agree to had plugin, we just need to decide what type and how to limit bruden for gitea and plugins developer by starting a roadmap and defining interface and what a plugin can do.

[lafriks]

I agree that there could be plugins for webhooks/ci integration but for extending core functionality and UI it will be too much work to implement that correctly, especially to support multiple platforms etc. Also if plugins starts to be created that adds UI functionality we will start to get bug reports that will be hard to track down if bugs are only reproducible with plugin or combination of plugins enabled.

I think that we should continue to keep Gitea easy to install, run and packed everything is single binary. To make plugins right it would mean to rewrite gitea core to allow that and most probably it would be quite big task with no direct benefit.

[tboerger]

I just want to mention that we already had #661 ;)

[tboerger]

Now my two cents for this topic. I wish we had an grpc based plugin system that is comparable to packer by hashicorp. Of course it's a lot of work, but we can even simplify the core and extract more parts into plugins. To avoid problems with plugin versions we have to apply versioned apis.

When we built the basics we can even offer some kind of marketplace where everybody can offer his stuff.

[jonasfranz]

@tboerger Will the plugins be compatible to newer / older "versioned apis" if gitea updates and the plugin not?
Which parts of the core should be extracted into plugins?

[tboerger]

@tboerger Will the plugins be compatible to newer / older "versioned apis" if gitea updates and the plugin not?

That's the whole thing of versioned APIs, older plugins should work with newer core versions, they just can not use newer core API parts.

Which parts of the core should be extracted into plugins?

No fixed list, maybe wikis, timetracking or other future additions like pages and so on. Keeping the core minimalistic and extracting many things into plugins should bring in some additional overhead, but every feature/plugin can be developed and published independent of the core as long as the core already provides the API for it.

[jonasfranz]

I think that a concept like drone's plugin system might be a good idea. But we need a better API since we do not only provide "build steps". So my idea is that every plugin is started as docker container and connected to the plugin API via gRPC like @sapk pointed out (#2222 (comment)). There is no installation process for the admin like JDK installation or other dependencies. But perhaps is this concept a little bit to complex.

[daviian]

Since we don't depend on Go 1.7 anymore, at least I thought so we could use the introducted go plugin architecture from 1.8.

A blogpost as an example: https://medium.com/learning-the-go-programming-language/writing-modular-go-programs-with-plugins-ec46381ee1a9

[lafriks]

@daviian we can not use built-in golang plugin architecture as it is not supported on Windows and other gitea build targers

[lunny]

@daviian we should declare v1.3 only support go1.8+ on README or something place.

[tboerger]

I would use some plugin system like packer, that's also grpc based and works also on Windows.

[sapk]

I could try to make a test with https://github.com/hashicorp/go-plugin. Any idea of a plugin that could be implemented ?
I was thinking of adding a file interpreter like markdown.

For the plugin architecture, I was thinking of allowing plugin to declare file handling based on regex/ file extension, declare url to be able to responds, declare event to listen for, be able to override some defined function.

[lavvy]

@sapk There are a ton of plugins here http://gitbucket-plugins.github.io/

[techknowlogick]

Another plugin system we could consider is the one that Caddy uses: https://github.com/mholt/caddy/wiki/Extending-Caddy

For the webserver for the docs site it is already using Caddy and its plugin system: https://github.com/go-gitea/gitea/blob/master/docs/Dockerfile#L8

[tboerger]

A compile time plugin system is not something I really like

[jonasfranz]

I've tested the built-in plugin system of golang and discovered that it might fit very well in our use case. Plugins will be using the already existing database functions in models without having a seperate plugin api interface.

Plugins are going to be ~39MiB big since they include the gitea sources. This is a problem in my mind.
Any ideas on this?

[tboerger]

The builtin plugin system doesn't support windows, or has that changed?

[Fardinak]

Plugin support has indeed improved since 1.8, but Windows is still unsupported at this time and it doesn't seem like there's any active effort towards making it happen. However the community seems keen on accepting a pull request from someone who knows actually how to implement it…

Plugin support in v1.13

Currently plugins are only supported on Linux and macOS.

Plugin support in master

Currently plugins are only supported on Linux, FreeBSD, and macOS.

And for reference, here's the issue regarding Windows support: golang/go#19282

Personally I'm mostly in favor of golang plugins or some sort of scripting. RPC just seems like too much trouble to manage. But in any case, I'm looking forward to plugin support.

[zeripath]

Oops, didn't mean to close this.

One thing I think would help would be a more extensible markdown renderer. I understand Hugo has moved to the goldmark renderer.

[tboerger]

I still think it would make sense to implement the plugin api based on grpc and use the hashicorp plugin lib, this is already heavily battle tested.

[lunny]

https://github.com/hashicorp/go-plugin allow only an executable binary plugin and running on the local network. So the plugin have be run on a sandbox(i.e. docker) to avoid it read/write filesystem/database directly.

[sapk]

@lunny I also think that the grpc solution is good because it allow local and remote. If plugin dev choose to acces database and file directly when run locally it is their concern. Maybe later we could choose to isolate (chroot, container, ...) local plugin but I think it would be extra first since it would be platform dependant.

[lunny]

@sapk I don't think that's their concern. I think the security should be considered when we design the plugin system.
Once you choose to enable plugin system, gitea could require you installed docker or k8s. All plugins should declare their permissions(read disk/visit network and other etc.)

[tboerger]

I don't get your concerns... That doesn't make sense to me.

[lunny]

@tboerger Plugins will run on your gitea server machine, it can do anything that's what my concerns. Could you explain why did you think that's not a problem?

[tboerger]

They can do whatever they implement... That's the use case for plugins 😂

[zeripath]

Depending on the level of plugin integration, there is a slight issue to do with plugins being run whilst there is an ongoing transaction with the db - but we'd just have to be explicit about when this happens. I would argue that level of integration would be too much to begin with.

If a server owner wants to blow themselves up by having a plugin that erases their entire db that's their call. We should however, try to proffer reasonable interfaces to the plugins to allow them to behave in a safe manner - so for example, offer sensible sanitisation services, perhaps offer some kind of limited js framework or even just best practices, or make it explicit which plugins extra js on the server - I dunno - I think the idea needs some concrete fleshing out to see what would need extra thought.

One thing that would make very tight plugin integration possible would be to move to a (configurable) dependency injection framework (There would be numerous benefits from this - including making it easier to test gitea properly). Go Wire is preferred by the go developers but I can't see how to make that make sense in a very configurable / plugin dominated system - something like a java Spring framework configurable file makes for very configurable system albeit with the costs of domain specific language. Again I dunno - just thoughts.

[tboerger]

Dependency injection doesn't exist in go... How to use wire you can see within drone.

Define some hook endpoints via interfaces, there drone could also give some advice. Start with first parts that can be extended and increase the hooks step by step

[Fardinak]

I think @lunny's concerns are valid, but that problem also exists with Go plugins or most other methods. I don't find requiring docker to be a viable option for a user that needs only a couple of plugins with their instance either. Also since OS support is clearly important, we need to remember that Docker still requires running a resource intensive VM on macOS.

Running plugins in a safe sandbox would be ideal. I consider WASI to be the ultimate solution, but unfortunately it just doesn't seem to be there yet. The closest option would be one of the interpreters mentioned by @andreynering:

About the plugin system, it's also hard to implement. Go has interpreters for other languages we could look:

* JavaScript: https://github.com/robertkrimen/otto
* Lua: https://github.com/yuin/gopher-lua
* Lua: https://github.com/Shopify/go-lua
* Bash: https://github.com/mvdan/sh (this one may not be suitable for the plugin system, but for allowing running bash commands that will run on Windows, too)

I must say, otto looks kinda promising.

Otherwise, it seems that GRPC is the most solid solution currently available. Paired with supervisord and it's RPC API (or maybe a golang implementation of it), they should provide a robust plugin solution for Gitea.

[andreynering]

Yaegi (a Go interpreter written in Go) is also a thing nowadays.

[Fardinak]

Oh that looks really interesting. How feasible would it be to create a limited subset of the stdlib? Although I don't know how that would affect the rest of the libraries, it could for example help disable the plugins from accessing the filesystem, or limit them to only the interface provided by Gitea.

[tboerger]

With all the concerns you should continue with the mono binary. I will unsubscribe from this issue because I currently don't believe this will lead to something useful.

[Fardinak]

Well it seems like the GRPC option makes sense to most. Let me know if I can be of help in any way. We're currently using Go+GRPC on a company project so I have some experience with it.

[mewalig]

Most of comments thus far seem to assume that the solution must be for gitea to dynamically load a module which is not incorporated into the original build of the gitea binary. That seems to have few advantages and many disadvantages, at least as a starting place.

What about just providing a format for modularizing the customization code, which allows for automating the integration of that code into the build process?

So instead of:

1. Build your gitea binary without customizations
2. Create your plug-in code, and build the plug-in binaries
3. Start gitea in some way that dynamically loads the above (assuming you can get this step to work)

you just do:

1. create your plug-in code, in some specific and known format (maybe this involves specific naming conventions, file + code snippet organization, etc)
2. Run a generic "merge" script that merges the plug-in code with the base code to generate composite code
3. Build your gitea binary from the composite code, creating a gitea binary with your customizations directly compiled in

This shifts the problem/challenge from dynamic loading, and all of the associated difficult problems, to a much simpler problem, which is to modify the build process to automatically merge the plug-in code with the base code, to generate the code that is ultimately compiled into the binary. This is not very hard, especially when one can easily define the initial scope to be as narrow as desired, and can also define the organizational structure of the new code to be whatever form is easiest to merge. Furthermore, this can be fairly easily done initially with plain text utilities, and while a more sophisticated approach to merging go code is not particularly trivial, it is also not that hard.

[lafriks]

This could be an option: https://scriggo.com/

[mewalig]

I have a budget of a few hundred thousand USD for developing this gitea code base further, and that number is likely to go higher, and that does not include a decent amount of $ that has already been spent. It's not a huge sum of money, but it can certainly fund a heck of a lot of the requested improvements that have not moved very quickly on this project.

We have already implemented several features that have been requested but not implemented in this code base, and we'd be interested in contributing portions of our changes to this code base, if for no other reason than to benefit from the shared maintenance burden and to give back a bit.

However, the lack of a plug-in system is making it challenging for us to do that in a manner that is not cost-prohibitive.

Any thoughts / suggestions as to how this can be moved forward?

[techknowlogick]

We have already implemented several features that have been requested but not implemented in this code base, and we'd be interested in contributing portions of our changes to this code base, if for no other reason than to benefit from the shared maintenance burden and to give back a bit.

Even if it is a PR with your current code that may have merge conflicts that would be beneficial for the public good, or even just a description of what you have done and how you have implemented it would be good too.

As it seems you have invested $$ of your engineering time into building stuff for gitea for yourself, perhaps funding Gitea maintainers directly for maintenance of Gitea would be beneficial as well, so that you can ensure continued development and support of Gitea. Several of us have Github sponsors, liberapay's, and patreons listed in our profile.

Any thoughts / suggestions as to how this can be moved forward?

Several of us have previously engaged in contract work to expand/maintain Gitea if there is a specific contract you'd like to discuss my email is in my profile, although I can also connect you with others who may be interested as well.

[mewalig]

My company has made $ contributions to the project but we'd be more inclined to do so in larger numbers if there could be a prior understanding or discussion as to how those funds get used. I've tried some of the bounty websites and so far my experience with them has been "offputting", to put it mildly. Do the existing funding options allow bounties to be placed on specific feature requests? I have not seen that but maybe I'm missing.