Make agent usable for external backends

[qwerty287]

Currently, you can extend agents with custom backends using go plugins (addons).

With #3268, this is migrated to RPC.

I don't think it's a good idea to use hashicorp go-plugins in the agent because they us RPC, but the agent uses gRPC too. So it's kinda duplicated.

Thus, I had the idea in this PR: Allow to import the agent as external package and run it with custom backends:

package main

import (
	"go.woodpecker-ci.org/woodpecker/v2/cmd/agent/core"
)

func main() {
	core.RunAgent(<your backend>)
}

This takes care of everything else and allows you to easily develop your own backend.

This replaces the addon approach for backends.

[anbraten]

Instead of compiling all backends (currently docker, local, kubernetes) into one agent and allowing to select at runtime using env var, only compile one backend which is then automatically executed.

I am not sure about that point. Sure it would be a bit smaller (afaik it isn't that large anyways), but the maintenance overhead for us (building, releasing, adjusting them, ...) separatley and for instance admins (downloading the correct agent, upps I got the wrong agent ...) could be expected to increase quite a bit.

I don't think it's a good idea to use hashicorp go-plugins in the agent because they us RPC, but the agent uses gRPC too. So it's kinda duplicated.

However both connections use totally different settings, scopes etc. I think we could allow custom agents to reuse the official agent code as a lib they can hook into, but would leave the standard agent as it is.

[qwerty287]

I am not sure about that point. Sure it would be a bit smaller (afaik it isn't that large anyways), but the maintenance overhead for us (building, releasing, adjusting them, ...) and for instance admins (downloading the correct agent, upps I got the wrong agent ...) could be expected to increase quite a bit.

Fine, I can undo it (if more people respond).

However both connections use totally different settings, scopes etc. I think we could allow custom agents to reuse the official agent code as a lib they can hook into, but would leave the standard agent as it is.

So you say it would also be fine to connect the backend via RPC to agent?

[anbraten]

So you say it would also be fine to connect the backend via RPC to agent?

That could also work, but I was mainly referring to the idea that the agent could also be use a lib. Where someone could add their own backend as well.

Apart from that are there currently any attempts to add a custom backend?

[qwerty287]

Apart from that are there currently any attempts to add a custom backend?

Not that I know of, but we have many open issues with new backends used only by a few users.

[woodpecker-bot]

Deployment of preview was successful: https://woodpecker-ci-woodpecker-pr-3270.surge.sh

[codecov]

Codecov Report

Attention: 6 lines in your changes are missing coverage. Please review.

Comparison is base (5cedc95) 35.74% compared to head (485b6de) 36.31%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
pipeline/backend/docker/docker.go	0.00%	2 Missing ⚠️
pipeline/backend/kubernetes/kubernetes.go	0.00%	2 Missing ⚠️
pipeline/backend/local/local.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3270      +/-   ##
==========================================
+ Coverage   35.74%   36.31%   +0.56%     
==========================================
  Files         227      223       -4     
  Lines       15107    14796     -311     
==========================================
- Hits         5400     5373      -27     
+ Misses       9308     9031     -277     
+ Partials      399      392       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[qwerty287]

@anbraten I removed the part to only support one backend. It's possible now to create an own custom agent with as many backends you want, selectable via env var.