Add plug-in framework to allow for configuration of lookup functions without hard-coding them into Hiera #35
Comments
|
@thallgren - this would be great, is there any timescale on implementation? I'd like to add another lookup function, should I wait for the plugin framework to be available? |
|
@hbuckle please don't let this hold you back. We can move things out again once we have a mechanism to do that. For now, I think it's fine to just add new functions to this repo. |
|
+1 to not letting perfection be the enemy of progress 😁 |
This commit moves the terraform_backend and azure_key_vault lookup functions from the Hiera binary and into their own repositories and adds a new `PluginLoader` that enables Hiera to be extended by regular Go Plugins. # Relates to lyraproj#35
|
I've created a new plug-in infrastructure in Hiera. In essence, it allows a "plugindir" to be added to the hiera.yaml with an additional "pluginfile". Both are optional and will default to the directory "plugin" (adjacent to the hiera.yaml) and ".so". When a lookup function is missing from the Hiera binary, it will use a special PluginLoader, configured with these values to load a Go plugin. It is assumed that this Go plugin will add the functions to the loader. I created a working set consisting of the "hiera" binary and two plugins, "hiera_terraform" and "hiera_azure" in my own Github namespace. The README.md files are of interest as they explain how things are configured. thallgren/hiera I'm aware that the names of the plugin repositories aren't ideal and subject to discussion. Perhaps we should have a "hieraplugin" organization or similar? An interesting observation: A docker image built with the hiera binary and the terraform plugin is 103 MB. Without the terraform plugin, it's just 23.4 MB. That alone I think is a good motivator for a plugin infrastructure. |
|
@thallgren This sounds great, the only issue for us is that go plugins don't work on Windows which is our main platform. Are there any alternatives? Hashicorp's go-plugin is one that I know of. |
|
@hbuckle I suspected that the lack of windows support would be a problem. If that's your platform, then it must to be resolved somehow, before this can be put to work. Go does have some support for loading dll's and I know there is ongoing work to integrate that into the plugin package. I'll investigate this further. The Hashicorp go-plugin is out of process which introduces a new set of problems and third-party dependencies that I would like to avoid if possible (we used it frequently in lyra). Also, such an external plug-in would become something in between the in-process plugin and the already existing RESTful service. One approach would be to enable building a RESTful service that just serves up one specific Hiera function and then load it in a similar fashion as the plugins. Doing so would mean that a plug-in could be a docker image and could be reached by a URL. |
|
@hbuckle I've now added lyraproj/hierasdk which is intended to be the only repository that a plugin will need. I have also changed the thallgren/hiera, thallgren/hiera_terraform, and thallgren/hiera_azure to use what that repository provides. This automatically turns them into RESTful plugins that are usable by the Hiera executable. I'd be very interested in your opinions on this and also if you could take it for a test spin to discover any issues with this approach in your Windows environment before I move this stuff up to lyraproj. In order for things to work, you'll need to use the branch "enable-plugins" in the thallgren/hiera repository and use master for all the others. |
|
@thallgren just done a quick test and it looks to be working perfectly:
I'll try things out a bit more today, but this is looking great. Thanks! |
This commit moves the terraform_backend and azure_key_vault lookup functions from the Hiera binary and into their own repositories and adds a new `PluginLoader` that enables Hiera to be extended by regular Go Plugins. # Relates to lyraproj#35
This commit adds a special plugin loader that kicks in when a lookup function is missing. The loader will make an attempt to load a RESTful plugin with the same name as the function from a "plugin" directory. The location of the plugin can be altered by configuring the "plugindir" and "pluginfile" attributes in the hiera.conf file. The commit removes the current terraform and azure-vault plugins from the hiera code-base. They are instead made available as plugins in repositories of their own. See: https://github.com/lyraproj/hiera_terraform https://github.com/lyraproj/hiera_azure Closes lyraproj#35
|
Barring objections, I intend to merge #38 tomorrow, Thursday. |
Hiera finds configured lookup functions (providers) using a loader. New functions can be added using the pcore
px.NewGoFunction(). This will however require that the function is compiled together with the rest of the Hiera binary (#34 is a good example). A better approach would be if such functions were reachable using some plug-in framework that could use a Hiera SDK and then made available as Go plugins (shared libraries / same address space) or as RESTful binaries (see #32).Adding the Azure lookup function alone increased the go.sum file from 75 to 227 lines, corresponding to a transitive dependency tree that is three times as large. It can be expected that other types of lookup functions will cause a similar increase in dependencies.
The text was updated successfully, but these errors were encountered: