ECS Compatibility - backport 7x #12308
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💚 CLA has been signed |
Implements a plugin `ecs_compatibility` option, whose default value is powered by the pipeline-level setting `pipeline.ecs_compatibility`, in line with the proposal in elastic#11623: In order to increase the confidence a user has when upgrading Logstash, this implementation uses the deprecation logger to warn when `ecs_compatibility` is used without an explicit directive. For now, as we continue to add ECS Compatibility Modes, an opting into a specific ECS Compatibility mode at a pipeline level is considered a BETA feature. All plugins using the [ECS Compatibility Support][] adapter will use the setting correctly, but pipelines configured in this way do not guarantee consistent behaviour across minor versions of Logstash or the plugins it bundles (e.g., upgraded plugins that have newly-implemented an ECS Compatibility mode will use the pipeline-level setting as a default, causing them to potentially behave differently after the upgrade). This change-set also includes a significant amount of work within the `PluginFactory`, which allows us to ensure that pipeline-level settings are available to a Logstash plugin _before_ its `initialize` is executed, including the maintaining of context for codecs that are routinely cloned. * JEE: instantiate codecs only once * PluginFactory: use passed FilterDelegator class * PluginFactory: require engine name in init * NOOP: remove useless secondary plugin factory interface * PluginFactory: simplify, compute java args only when necessary * PluginFactory: accept explicit id when vertex unavailable * PluginFactory: make source optional, args required * PluginFactory: threadsafe refactor of id duplicate tracking * PluginFactory: make id extraction/geration more abstract/understandable * PluginFactory: extract or generate ID when source not available * PluginFactory: inject ExecutionContext before initializing plugins * Codec: propagate execution_context and metric to clones * Plugin: intercept string-specified codecs and propagate execution_context * Plugin: implement `ecs_compatibility` for all plugins * Plugin: deprecate use of `Config::Mixin::DSL::validate_value(String, :codec)`
yaauie
force-pushed
the
ecs_compatibility_backport_7x
branch
from
ba73478
to
e11b6d3
Compare
|
jenkins test this again please |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR is a clean backport of #12305 to 7.x:
Why is it important/What is the impact to the user?
This is a part of an ongoing effort to add ECS Compatibility modes to Logstash plugins (#11635), and to have them on-by-default in the next major release of Logstash (#11623). The newly-introduced
pipeline.ecs_compatibilityallows a user to explicitly control the default behaviour of all plugins in a pipeline.This allows a user on Logstash 7.last to "lock in" a specific behaviour in advance of upgrading to Logstash 8.
Checklist
How to test this PR locally
The only plugin with ECS Compatibility that is currently shipped is the Elasticsearch Output, whose behaviour is changed to use ECS-compatibile index templates. Configure a pipeline that uses it, and run Logstash with
--pipeline.ecs_compatibility=v1set (command line) orpipeline.ecs_compatibility: v1(config, eitherlogstash.ymlor under a specific pipeline inpipelines.yml). Observe ECS-compatible templates being sent to Elasticsearch at startup.