Review Elasticsearch clients support

[bclozel]

As seen in #9379 and #12600, we need to consider our options for ElasticSearch clients and whether we should deprecate the native client since Spring Data ElasticSearch will support the high level rest client in the next Moore release train.

[mp911de]

Please note that High-Level REST Client support is sort of experimental right now. With Spring Data Moore we're going to evolve REST support to be production ready. Once this is in place, we will consider deprecating the Transport Client so we can remove it in Moore+1 or Moore+2.

[mp911de]

For Boot 2.2, we should deprecate the transport client and rather use the HTTP-based client. Elasticsearch 3.2.0 will contain also a reactive ES client (using WebClient) and reactive repositories.

[hantsy]

Spring Boot 2.2 M1 lacks auto-configuration for reactive data Elasticsearch, currently I have to enable it maunally. And the latest Spring data Elasticsearch(Moore) has included reactive repository and template APIs.

[wilkinsona]

@hantsy Thanks, but that's not really related to what's being tracked here. I opened #16214 to track the new auto-configuration.

[bclozel]

After reviewing the current situation, I think Spring Boot 2.2.0 should do the following:

1. spring-data-elasticsearch currently has a hard dependency on org.elasticsearch.client:transport. In Spring Boot, it is required to set the "spring.data.elasticsearch.cluster-nodes" configuration property in order to get a transport client. We should add configuration metadata to deprecate that property and let developers know that this support is deprecated now. In the future, spring-data-elasticsearch is likely to drop that transport dependency and this will make it disappear from the starter.
2. spring-data-elasticsearch also depends on org.elasticsearch.client:elasticsearch-rest-high-level-client, and we currently don't auto-configure an ElasticsearchOperations bean. See Auto-configure Elasticsearch REST client in Spring Data #16542
3. Since Moore, Spring Data Elasticsearch has a ReactiveElasticsearchClient and ReactiveElasticsearchOperations. We should support that use case. See Auto-configure reactive Elasticsearch components introduced in Spring Data Moore #16214
4. We should keep the plain RestClient / RestHighLevelClientConfiguration auto-configurations as they are needed for 2).
5. Given the numerous HTTP clients available now and the fast release pace of Elasticsearch, I'm wondering if we should deprecate Jest support as well.

[dadoonet]

@bclozel some food for thoughts about point 5:

• The Rest High Level client is now feature complete in 7.0 and 6.7.
• It is still coming with some dependencies that we will be cleaning up in the future to avoid pulling the whole elasticsearch core and its deps (like Lucene).
• I'd recommend using the official Rest High Level client from now as it's part of the same github than the elasticsearch server, meaning that every new feature is added and the code is highly tested.

The only drawbacks I can see for now is that you can't use a 7.x client with a 6.x server. Which means that Spring would have to provide the right "client"... Making things a bit complex may as per my personal project here.

[bclozel]

Thanks @dadoonet !

Spring Boot is managing elasticsearch dependencies to avoid mismatches in applications - developers can choose the elasticsearch version by changing a property in their build.

We'll have to make tough calls when it comes to choosing the right Elasticsearch version when upgrading. We're always in touch with the Spring Data team and we're following closely Elasticsearch's roadmap. For Spring Boot 2.2.0, we'll probably stick to Elasticsearch 6.7 and upgrade to 7.0 when the whole ecosystem has caught up.

[dadoonet]

Thanks @bclozel for the link to the documentation. We are starting to get a lot of questions about this problem in our forums, so I'm now linking to it to explain the workaround.