Identify existing integration with external clients and patch which are needed for 2.0.1 release

[dreamer-89]

Summary

As part of this issue, we need to identify integration with external clients which might currently be broken with OpenSearch 2.0. Integration can be broken because of how clients interpret engine version(beats) or communication mechanism(logstash). Once integration points are identified, patch the ones which are important.

Background

As part of #3484, we identified that OpenSearch 2.0 integration with external clients (filebeat) is broken.

Root cause

Lib-beat (core library providing ES APIs) have version checks where OpenSearch 2.0 is identified as ES = 2 and _type info is appended in bulk & search requests. As types were removed from OpenSearch 2.0, engine responds with illegal_argument_exception.

Existing state

The upstream maintainers don't have plans to support OpenSearch. This broken integration with beats is currently alleviated by ignoring _type in the bulk request meta data and temporarily adding the typed endpoints back for certain REST APIs. We also version spoofing [2] which should help with version comparison related issues.

Next steps

Issues which arise due to version mismatch can be solved by alternative #2. There are other use cases where patching on engine is needed for client support. We need to identify remaining integrations with external clients and fix the ones needed for upcoming patch release. Existing issues reported:

1. Elastic beats --> The pipeline processors have ES version checks. These checks might probably be related to pain for users of different pipeline processor types in filebeat.
2. Fluentbeat -> [BUG] java.io.OptionalDataException error in OpenSearch 2.0 rc security#1961 -->
3. LogStash
   i. https://forum.opensearch.org/t/default-index-mapping-doc-change-to-doc/7899 -> Adding back typed end-points may help.
   ii. [BUG] java.io.OptionalDataException error in OpenSearch 2.0 rc security#1961
4. log4j2 plugin -> Support OpenSearch 2.x and Elasticsearch 8.x rfoltyns/log4j2-elasticsearch#81

Alternative

1. [RFC] REST API Versioning #3035 --> Will take time
2. Revert "[Remove] MainResponse version override cluster setting (#3031) (#3032)" #3530 --> Version spoofing, will help only where version comparison is underlying issue.

[dreamer-89]

Discussed this internally with the team. Integration with external clients is already covered as part of #3532 and performing integration with all external clients is not possible. Instead we plan to track breaking integrations/issues via bugs opened on repository. Closing this issue in favour of #3532.