Http2 support for the Kibana server

[ppf2]

Configuring F5 Load balancer in front of tribe nodes for HA for Kibana. Would be nice for the Kibana server to support http2.

Tasks

Beta Give feedback

1. Add HTTP2 support Add http2 support for Kibana server #183465
2. Enable HTTP2 support for cloud/serverless (private issue)
3. Performance / load testing of Kibana between HTTP1 and HTTP2

Follow-ups

Beta Give feedback

1. Decide what to do with bfetch now that we have http2 optional support
2. Figure out our strategy regarding FTR coverage for HTTP2

[rashmivkulkarni]

@elastic/kibana-operations - any idea on what to do with this ? I have added enhancement label.

[jbudz]

works for me, but we're a long ways out,

at a minimum we need to upgrade hapi, but our bundling patterns for http1 are anti-patterns for http2. to get the single pipelining benefits we'll need to remove concatenation, inline images, etc.

if anyone's stumbled on this we would love to hear more about your use case. i do think we should do it but i don't think the web's quite there yet put this at a high priority. in the interim a proxy local to kibana configured for http2 should be okay.

[Asgoret]

@jbudz hi! is there any workaround?

[jbudz]

Hey @Asgoret, no new progress for Kibana yet. A reverse proxy local to Kibana is probably the closest option currently.

[Asgoret]

@jbudz already in use in red hat version of EFK, but it don't work. If i disable H2 on cluster router all works fine.

[binlab]

any progress?

[watson]

FYI: We're upgrading to hapi v18 in #80468. That should be one less roadblock in order to support http/2

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[ppf2]

http2 support may also help with the large number of concurrent .js files we try to download that tend to stall because of the per-server limit in many common browsers of no more than 6 file downloads at a time.

[mshustov]

per-server limit in many common browsers of no more than 6 file downloads at a time.

In prod mode Kibana loads them only once. on the next visit, they are loaded from the browser cache.

[mshustov]

I took a stab adding http/2 support to Kibana. You can find some necessary changes and page load time benchmarks in this PR #123748

There are a few open questions left:

• to check compatibility with a request proxy in Console app
• to verify whether switching to http/2 will affect other Stack components interacting with Kibana: fleet, filebeats, etc.
• check whether it's possible to use outdated superagent and supertest in the integration tests or they need to be updated to the latest versions with http/2 support
• perform load testing to measure impact on the Kibana server.

It doesn't look like a lot of work, we can prioritize the work if we think the page load time is one of Kibana's bottlenecks.

[thomasneirynck]

tl:dr;

Moving to http2/3 would be a concrete improvement to Dashboard performance and improve the overall developer experience for teams that work on apps like Dashboard, Discover, Lens, Maps ("Analytics").

---

bsearch introduces overhead kibana server, and some minimal overhead on the client.

bsearch is a work-around to http1 connection limit. This work-around is useful for Dashboards.

bsearch collects requests on the client on a debounce-interval (introducing a small delay). On the server, it fans out these individual requests to elasticsearh. It then serializes and concatenates these responses and sends them back as a single payload to the client. In turn, the client deserializes the responses back to the original Elasticsearhc response JSON.

The overall mechanism is explain here #166206

bsearch introduces a known overhead:

• CPU overhead on the server (due to re-encoding and concatenation of responses)
• CPU overhead on the client (parsing and decoding of these concatenated results)
• slight delay in data loading on a Dashboard (the debounce schedule to collect requests on the client)

bsearch degrades the developer experience

bsearch introduces additional mental load:

• bsearch obfuscates the server response, making debugging and troubleshooting harder.
• This Home-grown batching mechanism clutters the overall picture of how a dashboard functions and makes it harder to reason about its overall performance characteristics.
• Adds additional overall overhad to maintain this code.

bsearch blocks incremental progress wrt data format

Switching to one of the more compact output formats supported by Elasticsearch, like smile or CBOR, would be a small, concrete win to reduce the overall network traffic to Dashboards.

Some of these benefits are wasted on the client<->kibana-server connection if bsearch remains as a middleware. We would still need the overhead of the reserialization and concatenation of responses.

It would also require additional investment in bsearch, something we likely no longer want to do (see all other reasons)

working towards a thin kibana server

The removal of bsearch is the most concrete contribution we can make for apps like Discover, dashboards, lens, or maps to reduce their load on Kibana Server.

anticipating streaming, webgl, and binary formats

in the future, we are likely seeing a move to data formats with higher data density, and which have better characteristics for streaming (such as arrow), to support more advanced chart types (e.g. scatterplots). An intermediate batching layer which re-encodes data does not fit into this model.

code and asset loading benefits

We would also expect a broader improvement wrt code and asset loading, something which would benefit all apps, not just Dashboards.

[TinaHeiligers]

Do elasticsearch and kibana's proxy support HTTP2?

[thomasneirynck]

@TinaHeiligers , wrt #7104 (comment)

Do elasticsearch

no. There is some discussion here elastic/elasticsearch#10981 .

imho - there's less relevance for the kibana<-->ES connection to be http2, because it's usually (hopefully! ;)) not a browser that connects directly to ES.

and kibana's proxy support HTTP2?

I believe yes. To see the benefits in a dashboard, it would be important to ensure that the proxy is http2 as well.

[lukeelmers]

and kibana's proxy support HTTP2?
I believe yes.

Correct - the proxy supports http2, and currently requests to kibana are downgraded to http/1.1, which we would stop doing once Kibana supports http2.

[pgayvallet]

I recently resurrected @mshustov's POC to open my own investigation PR.

The outcome is unsurprisingly the same as it was 2 years ago: if booting Kibana with a http2 listener is relatively easy, making sure that everything works fine with it, that our tooling (test tooling mostly) supports it, and that other components of the stack are able to communicate with Kibana when http2 is enabled will be most of the work in that initiative.

Things to know

HTTP2 "requires" TLS

Even if it's not a requirement in the spec, all browsers will only accept to use HTTP2 against a secure server.

There are two main reasons for that:

• it was a good way to force everyone to go more secure (yeah)
• protocol negotiation is done using ALPN, and ALPN is a TLS feature (we'll get back on that)

Note that, as said, this is not a requirement on the spec, and it is possible to perform unsecure http2 communication between two servers (or more commonly, a proxy and its backend), as long as the two components are fine with it and don't need protocol negotiation.

ALPN and protocol negotiation

ALPN (for Application-Layer Protocol Negotiation) is a TLS feature extension that allows to negotiate the protocol that will be used over the connection.

Basically, in our scenario, this is what will allow to downgrade connections performed against an HTTP2 server to HTTP/1 if the client doesn't support HTTP2. This is very important (not to say mandatory) for the component serving the public traffic, and is why we will absolutely want/need to have TLS configured on Kibana when http2 is enabled (most of the time)

Things we will have to do

Implement HTTP2 support

Obviously the first stage. #183465 is doing most of the work, but we need to polish it and get it merged.

I don't think we need a feature branch for it (given http2 will be disabled by default), but we might want to lock the protocol option under dev mode until we're fine releasing it as experimental.

Confirm components of the stack are able to communicate with Kibana when HTTP2 is enabled

So in theory, with ALPN (so TLS) enabled and accepting downgrades to HTTP/1.x, nothing would need to be done and everything should work "out of the box" if we enable HTTP2 on the Kibana server.

However, we will have to validate that.

Components that I can think of (list likely not exhaustive)

• The Kibana health gateway server
• Fleet
• Filebeat
• ???

Adapt our http clients to use HTTP2

So, there are two parts of that:

Note that in theory, as long as protocol negociation works, this shouldn't be blocker for an "experimental release" of HTTP2 support, as all those clients should gracefully downgrade to HTTP/1 (but again, we will need to confirm that)

Replacing agents that don't support HTTP2 at all

Axios for example has an issue open for more than 5 years, but it still not supporting http2 (and of course, we are using axios quite a bit in both our tooling and production code).

We will have to migrate to another http agent instead (seems like ky may be the way to go today?)

We need to also figure out which other(s) clients we might be using across our codebase

Enabling http2 on our agents (when http2 is enabled on the server)

This was quite of a surprise to me, but most (all non-browser?) agents are not supporting ALPN (or at least don't have http2 enabled out of the box). Meaning that if you want to use them against an http2 backend, you have to explicitly configure it that way at the agent or request level.

For example, this is how it works for supertest:

const request = require('supertest');
// request level
request(app, { http2: true }).get('/user');
// agent level
request.agent(app, { http2: true }).get('/user')

Note that performing a request with http2 disabled on the client-side against an HTTP2 server accepting downgrading via ALPN works, but it doesn't the other way (http2-enabled client against a plain HTTP1 TLS client), so we can't just always enable http2, and will have to do it only when the server is supporting it.

For test tooling, this implies some work, but is relatively straightforward, given we know we're testing directly against the Kibana server without middlewares or proxies.

For our production usages of clients (e.g. our production usage of axios), it may be trickier, given having http2 enabled on the server !== http2 available for the browser (depending on the custom infra), so we might need to think a bit of how exactly we want to manage that.

But again, with protocol downgrade, this should be doable as a follow-up enhancement.

Adapting our internal tooling

Dev mode

Do we want to enable http2 by default for dev mode? If so, we need to change the CLI script accordingly, to at least add the TLS dev certificate to the config.

Dev server

If we want to support HTTP2 with "virtual" base-path on dev-mode, we will have to adapt the base path proxy to accept http2 connection and proxy http2 connection. It may require to fully change on implementation

kibana/packages/kbn-cli-dev-mode/src/base_path_proxy_server.ts

Lines 131 to 137 in d893a06

	this.server.route({
	handler: {
	proxy: {
	agent: this.httpsAgent,
	passThrough: true,
	xforward: true,
	mapUri: async (request: Request) => {

FTR test runner / test configuration

This is a big one.

The first question to ask is what the strategy should be regarding http2 enablement for our functional test suites.

• Do we want to run all our tests only against an http2 server?
• Do we want to run all our tests against BOTH an http2 and an http1 server?
• Do we want to randomly enable http2 for some suites?
• Do we want just some smoke + some specific test suites using http2?

I personally feel the last one would be our best option, but we need to get to a consensus.

Then the second part is changing our FTR configuration accordingly. As explained previously, there are two major things impacting this protocol switch for FTR tests:

• we need TLS enabled for http2, so we need to configure the server accordingly (should be fine to use the dev self-signed certificate and configuring the browser to accept them)

• we need to configure our tools to use http2.

E.g. for supertest, we should be able to globally enable http2 at the agent level at the provider's level:

kibana/test/api_integration/services/supertest.ts

Lines 16 to 20 in d2feac0

	export function KibanaSupertestProvider({ getService }: FtrProviderContext) {
	const config = getService('config');
	const kibanaServerUrl = formatUrl(config.get('servers.kibana'));
	return supertest(kibanaServerUrl);
	}

Adapting our APIs

Config validation

Should we have the config fail validation if http2 is enabled without TLS? should we just log a warning?

Expose the protocol information internally

We already have a protocol field on HttpServerInfo, but it's not really what the name implies:

kibana/packages/core/http/core-http-server/src/http_contract.ts

Lines 395 to 404 in 364ee52

	export interface HttpServerInfo {
	/** The name of the Kibana server */
	name: string;
	/** The hostname of the server */
	hostname: string;
	/** The port the server is listening on */
	port: number;
	/** The protocol used by the server */
	protocol: 'http' | 'https' | 'socket';
	}

We will want to add the information of the protocol (http1 / http2) to this structure.

More important, given we will support protocol fallback, we probably want / need to be able to expose this information at the request level. The info is available on the raw requests (httpVersion, so we should be able to just mirror it)

Perform performance testing to compare behavior between the two protocols

In theory, enabling http2 should only be beneficial. However, we ideally need to confirm that and put numbers on it.

Conclusion

This is likely not an exhaustive list - I will edit/update it when needed.

[pgayvallet]

#183465 was just merged, bringing HTTP2 capabilities to the Kibana server. The feature is flagged as experimental, but will be available in Kibana 8.15 and higher.

The next two major tasks are:

• work with Cloud control plane team on the details regarding enabling HTTP2 support for serverless, and potentially later ESS.
• perform load testing of the Kibana server with and without HTTP2 enabled (but with same TLS config) to make sure there's no performance degradations when HTTP2 is enabled

In the meantime, I updated the issue with a "follow-ups" task list. At the moment, we have two items:

• Decide what to do with bfetch now that we have http2 optional support (cc @thomasneirynck)
• Figure out our strategy regarding FTR coverage for HTTP2 (cc @pheyos)

[thomasneirynck]

thanks @pgayvallet, fantastic news. yes, we will follow up. Our next step is to validate some of the improvements we are expecting (#183096), and looking at how we can phase the removal of bsearch for stateful (possibly tied into with 9.0 release). cc @lukasolson @vadimkibana @davismcphee @kertal