support HTTP/2.0

[inotnako]

Is ElasticSearch planning to add support (or maybe plugin for) protocol HTTP/2.0?
Like in https://github.com/grpc/grpc(used HTTP/2.0) or may be with grpc

[dakrone]

This is now possible since we have moved to Netty 4, since Netty supports HTTPP/2.0 in 4.1+ /cc @jasontedor

[jasontedor]

To be clear, Netty 4 supports HTTP/2 but there is effort required to make Elasticsearch use it, and ensure that it works properly.

[KodrAus]

Is there any new thinking on supporting h2? It seems like a fruit hanging so high it'll probably take a magic beanstalk to reach, but I'm interested in any thoughts you might have on the cost-benefit equation. I imagine there are bigger wins in first-class client support and alternative APIs that take advantage of more stream-oriented connections in supporting h2 within Elasticsearch itself over just exposing through a h2-enabled proxy.

[mantasindrasius]

Any updates on this? With the deprecated TransportClient, there should for sure be HTTP/2 or/and gRPC support.

[jasontedor]

At this time there is no update on this other than that we are not currently working on it; this not to say that we will not, only that we feel there are higher priority projects. I do not agree that the deprecation of the transport client implies that there should be HTTP/2 support. Can share why you think that the two are tied together?

[mantasindrasius]

Thanks for a quick answer.

HTTP/1.1 is limited in the sense that any concurrent operations cannot happen on the same connection and creating a new connection is a huge performance trade-off.

HTTP/2 or gRPC which is based on the former is multiplexing by nature, so is closer to the non-blocking TransportClient.

If we want to keep the transport overhead low, we can't go with REST+HTTP/1.1.

[jasontedor]

@mantasindrasius Thanks for the feedback. I agree with you on the benefits of HTTP/2.

I do think it's worth noting that with HTTP/1.1 keep alive behavior and a connection pool that you can mitigate some of the disadvantages of HTTP/1.1 versus HTTP/2.0. The best that HTTP/1.1 can offer related to multiplexing is pipelining. Yes, this suffers from head-of-line blocking but it does allow some connection reuse too.

That said, even weighing these advantages and disadvantages, I would prefer that we continue to not tie removal of the transport client to the introduction of HTTP/2. Removing the transport client will bring us major advantages in continuing to develop the server side of Elasticsearch without maintaining a client that is so intimately tied to the internal protocol. These advantages outweigh the loss of benefits of the transport client.

Rest assured that HTTP/2 is on our roadmap, just not at the top of the list at the moment.

[mantasindrasius]

Thanks for that. Do you also think there could be gRPC support for elastic or it's not on the table?

[jasontedor]

@mantasindrasius gRPC is not on the table at the moment.

[Asgoret]

@jasongoodwin ELK don't work in OKD 3.11 (Openshift) correctly because there is no support of H2\grpc protocols.
See: elastic/kibana#7104 and openshift/origin-aggregated-logging#1465

UPD#1: Yeah...i disable h2 and kibana work perfectly. Enable and stop working) Is there ant information when kibana will support h2?

[ayushsangani]

Any updates on the support of Http 2.0 in ES?

[cangiremir]

Any actions to support gRPC or HTTP/2?

[weitzjdevk]

Are there any news here? AWS NLB has finally ALPN support for the TLS Listener, which would help for the gRPC handshake/http2 connection

https://www.reddit.com/r/aws/comments/gsdo3w/nlb_now_supports_alpn_on_tls_listeners/

[bachden]

Waiting for http2 (gRPC maybe) supporting.

[bradquarry]

A team at Verizon is asking for gRPC support. +1

[YohanSciubukgian]

Do you have any update to share about HTTP/2 support please ?

[gokcesurenkok]

Do you have any plan to do that in near future ? i would like to see it if it is in your roadmap.
Thanks

[weitzj]

Any updates here?

AWS ALB has finally gRPC support built in
GCP had it all the time

[cyberfifi]

+1 from Compass. We need this feature.

[emcpadden]

This is definitely needed!

[Farenheith]

I endorse http2 too!

One solution for me at this moment, if anyone is interested, is to avoid elasticsearch oficcial client (in a nodejs service that only uses _search), and to use superagent + agentkeepalive, with, in my case of use, maxSockets uninformed, assuming the default value of Infinity.
This way I have the connection pool working under the hood, mitigating the http1 limitation as @jasontedor well said.

[gguillen]

+1 Upvote

[JhennerTigreros]

A team at Platzi is asking for gRPC support. +1

[Dyqer]

+1 Upvote

[Jaimenms]

+1 Upvote

[cjmcgraw]

+1 this would be very helpful to my company

[kkdeok]

+1 please

[marinrusu1997]

+1 Upvote

[KennyLindahl]

+1 Upvote

[BiditPakrashi]

Any update on GRPC support ?

[mhlee-lee]

+1 please

[fsedano]

+1 please

[nakulpathak3]

+1 Faire would like this too

[raff-s]

+1 Upvote

[DaemonSnake]

+1 Upvote

[yanchanglj]

+1 please

[vineelyalamarthy]

@clintongormley are there discussions happening some where like in wiki pages/ discord etc on this topic roadmap/implementation. Are open source contributors welcome to participate in those? I would love to contribute to this feature.

[lukasojd]

+1

[henningandersen]

This is not a high priority to us internally at the moment. There is no to little actual benefits from http/2 over what we have today. We would welcome a contribution, but would like to remind of our contributions guidelines, highlighting that this change is likely to require considerable test additions.

[otrosien]

@henningandersen thanks for the feedback. Out of curiosity: "There is no to little actual benefits from http/2 over what we have today" - how was that evaluated?

[loeffel-io]

+1

[magicleo]

Any update on GRPC support ?

[patpatpat123]

Upvoting this many year old idea

[firebook]

Any update? We really need it!

[DaveCTurner]

Unfortunately there's nothing in the discussion above to say why this is so important. HTTP/2.0 (and now HTTP/3.0) offer significant advantages for certain applications, particularly those involving web browsers or web-browser-like interaction patterns, but we cannot see the benefits over HTTP/1.1 in the specific context of an application like ES whose clients are expected to use a pool of long-lived connections (and possibly pipelining) to achieve the same results. We just don't encounter cases where the use of HTTP/1.1 is the limiting factor for either performance or expressivity in a meaningful way right now. I'm sorry if we're missing something obvious, but without some more concrete motivation for addressing this request it's unlikely we'll take on the significant amount of work involved in doing so in the foreseeable future.

[trycoon]

@DaveCTurner HTTP/2 is of little importance to me… But I was hoping that gRPC could lower the transmission latency that JSON and REST adds to the querying. We use Elastic as a smarty cache to filter information and answer user requests, several thousend per second. If we could shave of some additional 10ms for each 100KB JSON response by using gRPC, that would be a nice improvement.
And to be able to use gRPC we need HTTP/2… 🤗

[DaveCTurner]

If we could shave off some additional 10ms for each 100KB JSON response by using gRPC, that would be a nice improvement.

That would indeed be very nice, but I don’t see how it could happen. The current wire protocol doesn’t add anything like that much latency.

[mselmansezgin]

@DaveCTurner the wire protocol doesn't add anything for additional latency but it also detracts from the ability to use protobuf instead of JSON. This eliminates our chance to reduce the overall size of the payload.

[DaveCTurner]

Adding HTTP/2.0 support wouldn't itself yield any meaningful savings in this area because today ES needs the data it processes to use a self-describing format such as JSON (although if you care about payload size it'd be better to use one of the other supported formats instead). Protobuf isn't naturally self-describing in the way ES needs today, so we'd either end up adding loads of metadata or else just making a thin protobuf wrapper around one of the existing formats, neither of which would be meaningfully smaller than today's wire protocol.

We could probably teach ES to speak protobuf more naturally too, separating schema from content better, but that'd be a separate enhancement. If we did that it'd be able to speak protobuf over the existing HTTP/1.1 interface as well. In other words this point is pretty much orthogonal to the conversation about HTTP/2.0.

[nakulpathak3]

+1 HTTP/2 is extremely commonly supported in 2024

[DaveCTurner]

Thanks @nakulpathak3, I think my earlier message addresses this.

[swallez]

http/2 has ping frames, which can be sent by both client and server. Having Elasticsearch support http/2 would allow it to send periodic ping frames that would keep the TCP connection alive for long-running requests by transparently informing the client application that processing is still going on.

This blog post shows a concrete benefits of using them in the context of gRPC (gRPC keepalive is just http/2 ping).

Edit: it seems that proxies do not forward http/2 ping frames (whether this is correct or not), as shown here and here.