Rework the plugin loading logic

[dominiklohmann]

📔 Description

This PR contains a set of changes related to how we load plugins.

1. --plugins and --plugin-dirs options may now be specified on the command line as well as the configuration under vast.plugins and vast.plugin-dirs respectively.
2. We no longer load static plugins by default.
3. The CMake build option VAST_ENABLE_PLUGIN_AUTOLOADING no longer exists.
4. Adding bundled to vast.plugins causes all bundled plugins, i.e., static or dynamic plugins built alongside VAST, to be loaded.
5. Adding all to vast.plugins causes all bundled and external plugins to be loaded.

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

Review code file-by-file.

Regarding the design: I went over this with @mavam and we agreed this was a good UX improvement. I will wait until after the initial review with writing the documentation for this, though.

To test this change I recommend the following steps:

1. Test various combinations of the new plugin option with a bundled plugin in a standard location.

   cmake -B build.vast -D "VAST_PLUGINS=${PWD}/plugins/pcap"
   cmake --build build.vast
   cmake --install build.vast --prefix "${PWD}/install.vast"
   vast -v --plugins=... version

2. Test various combinations of the new plugin option with an external plugin in a non-standard location and a bundled plugin in a standard location.

   VAST_DIR="${PWD}/install.vast" -S example/plugins/analyzer -B build.analyzer
   cmake --build build.analyzer
   vast -v --plugins=... --plugin-dirs="${PWD}/build.analyzer/lib/vast/plugins" version

3. Test various combinations of the new plugin option with an external plugin in a non-standard location and a bundled plugin in a standard location.

   cmake --install build.analyzer --prefix "${PWD}/install.vast"
   vast -v --plugins=... version

[lava]

Just a quick first look.

Maybe you can expand the description a bit to include what the specific issues were that you and @mavam were trying to solve with this redesign, otherwise its hard to tell if it succeeds at that or not.

As a general feeling, it seems like expanding the bundled and all keywords into a list of plugins could be done outside the routines that handle the actual loading of plugins, to disentangle the code.

[dominiklohmann]

Maybe you can expand the description a bit to include what the specific issues were that you and @mavam were trying to solve with this redesign, otherwise its hard to tell if it succeeds at that or not.

A few things that were all interrelated:

• Do no longer enable static plugins by default to treat both types of bundled plugins equally.
• Make it possible to use --plugins= and --plugin-dirs= on the command line.
• Add an option to load all bundled plugins
• Add an option to load all installed plugins including external plugins

As a general feeling, it seems like expanding the bundled and all keywords into a list of plugins could be done outside the routines that handle the actual loading of plugins, to disentangle the code.

It cannot with the current behavior, unless all should also hard fail for libraries that cannot be loaded.

[mavam]

As a general feeling, it seems like expanding the bundled and all keywords into a list of plugins could be done outside the routines that handle the actual loading of plugins, to disentangle the code.

Mixing --plugins=all with --plugins=foo,bar,baz may confuse different things. Putting into separate options could be alternative, e.g.,

• Explicit plugins: --plugins=foo,bar,baz
• Plugin groups: --load-bundled-plugins or --load-all-plugins (modulo option naming)

[dominiklohmann]

• Explicit plugins: --plugins=foo,bar,baz
• Plugin groups: --load-bundled-plugins or --load-all-plugins (modulo option naming)

I'd like to avoid doing that, as adding more options that need special-case handling in the CAF configuration is a pain to maintain. I also think @lava was talking about the code itself more, where we can expand the all and bundled keywords into a list of plugin file in-place. I actually thought about this some last evening and came up with an idea of how to further refactor this with that in mind.

[lava]

Looks generally good to me; just a few stylistic comments below.

Also, I think it would be good to capture the motivation for this PR that you explained in a video call somewhere (maybe a new "feature" changelog entry?), since your previous answer just gave a summary of the PR contents.