[spacetime] Http/2 support #123748

mshustov · 2022-01-25T18:16:30Z

Summary

parent issue #7104
This PR attempts to estimate the effort of adding HTTP/2 support to the Kibana web server.
Note that the change won't improve load time on Cloud. Kibana browser app already connects to the Cloud proxy via http2. Cloud proxy converses http/2 into http/1 to communicate with the Kibana server though.

Why

HTTP/2 protocol is multiplexed, so the browser will use a single connection for parallelism. Kibana, in the current implementation, reforms about 200 requests to the Kibana server on start. Web browsers limit the number of parallel connections to six per domain. Therefore, the requests are stuck in a queue, and Kibana has sub-optimal UX for a high number of parallel requests to the Kibana server. To work the problem around we use different techniques like long-term caching and request batching. Both have critical problems. Caching might be disabled or periodically cleared on a client. Request batching adds additional overhead. Using HTTP/2 makes these workarounds obsolete.

TODO

to check compatibility with a request proxy in Console app
to verify whether switching to http/2 will affect other Stack components interacting with Kibana: fleet, filebeats, etc.
to verify APM agent supports http/2. It supports http/2 protocol https://github.com/elastic/apm-agent-nodejs/blob/main/lib/instrumentation/modules/http2.js
check whether it's possible to use outdated superagent and supertest in the integration tests or they need to be updated to the latest versions with http/2 support
perform load testing to measure impact on the Kibana server.

Performance impact

on the loading time

Tested on e2-standard-4 instance on gcp in europe-west2-a zone via lighthouse, landing on /home page
main branch

PR

Known problems

The full list of incompatibilities between http/1 and http/2 in node nodejs/node#29829

request performed by nodejs http agents without http/2 support (axios, for example) will fail. This PR doesn't fix all internal calls.
Base path proxy throws on proxying pseudo-headers https://github.com/nodejs/node/blob/master/lib/internal/http2/compat.js#L105-L109
Hapi attaches an unsupported connection header to error responses. I worked it around by disabling unsupported warnings in 03b7e52
bfetch can be refactored to support http/2. Maybe we don't need it at all and can fall back to the simple HTTP call? HTTP/2 doesn't suffer from the limitation of the number of parallel requests to a single domain. [Search] Benchmark bfetch impact in cloud with http2 #124538

mshustov · 2022-01-25T18:18:00Z

packages/kbn-std/src/pick.ts

@@ -7,11 +7,12 @@
 */

 export function pick<T extends object, K extends keyof T>(obj: T, keys: readonly K[]): Pick<T, K> {
-  return keys.reduce((acc, key) => {
-    if (obj.hasOwnProperty(key)) {


in the case of http/2, headers object is created via Object.create(null) so it hasn't got hasOwnProperty method

mshustov · 2022-01-25T18:18:22Z

packages/kbn-test/src/kbn_client/kbn_client_requester.ts

@@ -67,11 +66,11 @@ export interface ReqOptions {
  path: string;
  query?: Record<string, any>;
  method: 'GET' | 'POST' | 'PUT' | 'DELETE';
-  body?: any;
+  body?: Record<string, any> | string;


TODO: check whether we use string anywhere

This reverts commit faa3b40.

kibana-ci · 2022-01-27T16:48:45Z

💔 Build Failed

Failed CI Steps

Test Failures

[job] [logs] OSS Accessibility Tests / a11y tests using flights sample data "after all" hook in "using flights sample data"
[job] [logs] OSS Accessibility Tests / a11y tests using flights sample data "after all" hook in "using flights sample data"
[job] [logs] Default CI Group #2 / Advanced Settings security feature controls global advanced_settings all privileges "before all" hook for "shows management navlink"
[job] [logs] Default CI Group #2 / Advanced Settings security feature controls global advanced_settings all privileges "before all" hook for "shows management navlink"
[job] [logs] Default CI Group #17 / alerting api integration security and spaces enabled Actions "after all" hook in "Actions"
[job] [logs] Default CI Group #17 / alerting api integration security and spaces enabled Actions "after all" hook in "Actions"
[job] [logs] API Integration Tests / apis KQL telemetry API "after all" hook for "should only accept literal boolean values for the opt_in POST body param"
[job] [logs] API Integration Tests / apis KQL telemetry API "before all" hook for "should increment the opt in counter in the .kibana/kql-telemetry document"
[job] [logs] Default CI Group #18 / apis Monitoring Setup Collection detect_logstash_management "after all" hook: unload archive for "should get collection status"
[job] [logs] Default CI Group #18 / apis Monitoring Setup Collection detect_logstash_management "after all" hook: unload archive for "should get collection status"
[job] [logs] Default CI Group #18 / apis Monitoring Setup Collection detect_logstash_management "before all" hook: load archive for "should get collection status"
[job] [logs] Default CI Group #18 / apis Monitoring Setup Collection detect_logstash_management "before all" hook: load archive for "should get collection status"
[job] [logs] Default CI Group #11 / apis SecuritySolution Endpoints basic licsense cases feature privilege "after each" hook for "User sec_all_user with role(s) sec_all_role can create a case"
[job] [logs] Default CI Group #11 / apis SecuritySolution Endpoints basic licsense cases feature privilege "after each" hook for "User sec_all_user with role(s) sec_all_role can create a case"
[job] [logs] Default Firefox Tests / Canvas app "after all" hook: afterTestSuite.trigger in "Canvas app"
[job] [logs] Default Firefox Tests / Canvas app "after all" hook: afterTestSuite.trigger in "Canvas app"
[job] [logs] Default Firefox Tests / Canvas app smoke test "after all" hook for "renders elements on workpad"
[job] [logs] Default Firefox Tests / Canvas app smoke test "after all" hook for "renders elements on workpad"
[job] [logs] Default Firefox Tests / Canvas app smoke test "before all" hook for "loads workpad list"
[job] [logs] Default Firefox Tests / Canvas app smoke test "before all" hook for "loads workpad list"
[job] [logs] Default CI Group #27 / cases security and spaces enabled: basic push_case "after each" hook for "should get 403 when trying to create a connector"
[job] [logs] Default CI Group #27 / cases security and spaces enabled: basic push_case "after each" hook for "should get 403 when trying to create a connector"
[job] [logs] OSS Firefox Tests / console app "after all" hook: afterTestSuite.trigger in "console app"
[job] [logs] OSS Firefox Tests / console app "after all" hook: afterTestSuite.trigger in "console app"
[job] [logs] OSS CI Group #1 / context app "after all" hook in "context app"
[job] [logs] OSS CI Group #1 / context app "after all" hook in "context app"
[job] [logs] OSS CI Group #1 / context app "before all" hook in "context app"
[job] [logs] OSS CI Group #1 / context app "before all" hook in "context app"
[job] [logs] OSS Misc Functional Tests / core deprecations service Public API "before all" hook for "#getAllDeprecations returns all deprecations plugin deprecations"
[job] [logs] OSS Misc Functional Tests / core deprecations service Public API "before all" hook for "#getAllDeprecations returns all deprecations plugin deprecations"
[job] [logs] Jest Tests #5 / createPluginInitializerContext context.config config.globalConfig$ should be an observable for the global config
[job] [logs] OSS CI Group #6 / dashboard app new charts library dashboard state "after all" hook for "Saved search will update when the query is changed in the URL"
[job] [logs] OSS CI Group #6 / dashboard app new charts library dashboard state "after all" hook for "Saved search will update when the query is changed in the URL"
[job] [logs] OSS CI Group #6 / dashboard app new charts library dashboard state "before all" hook for "Overriding colors on an area chart is preserved"
[job] [logs] OSS CI Group #6 / dashboard app new charts library dashboard state "before all" hook for "Overriding colors on an area chart is preserved"
[job] [logs] OSS CI Group #2 / dashboard app using current data empty dashboard "after all" hook for "should open editor menu when editor button is clicked"
[job] [logs] OSS CI Group #2 / dashboard app using current data empty dashboard "after all" hook for "should open editor menu when editor button is clicked"
[job] [logs] OSS CI Group #2 / dashboard app using current data empty dashboard "before all" hook for "should display empty widget"
[job] [logs] OSS CI Group #2 / dashboard app using current data empty dashboard "before all" hook for "should display empty widget"
[job] [logs] OSS CI Group #3 / dashboard app using current data full screen mode "before all" hook for "option not available in edit mode"
[job] [logs] OSS CI Group #3 / dashboard app using current data full screen mode "before all" hook for "option not available in edit mode"
[job] [logs] OSS CI Group #5 / dashboard app using legacy data dashboard save "before all" hook for "warns on duplicate name for new dashboard"
[job] [logs] OSS CI Group #5 / dashboard app using legacy data dashboard save "before all" hook for "warns on duplicate name for new dashboard"
[job] [logs] OSS CI Group #4 / dashboard app using legacy data dashboard time picker "after all" hook for "Timepicker respects dateFormat from UI settings"
[job] [logs] OSS CI Group #4 / dashboard app using legacy data dashboard time picker "after all" hook for "Timepicker respects dateFormat from UI settings"
[job] [logs] OSS CI Group #4 / dashboard app using legacy data dashboard time picker "before all" hook for "Visualization updated when time picker changes"
[job] [logs] OSS CI Group #4 / dashboard app using legacy data dashboard time picker "before all" hook for "Visualization updated when time picker changes"
[job] [logs] OSS CI Group #10 / dashboard elements dashboard elements ciGroup10 input controls input control options "before all" hook for "should not have inspector enabled"
[job] [logs] OSS CI Group #10 / dashboard elements dashboard elements ciGroup10 input controls input control options "before all" hook for "should not have inspector enabled"
[job] [logs] Default CI Group #19 / dashboard feature controls dashboard feature controls security "after all" hook in "dashboard feature controls security"
[job] [logs] Default CI Group #19 / dashboard feature controls dashboard feature controls security "after all" hook in "dashboard feature controls security"
[job] [logs] Default CI Group #19 / dashboard feature controls dashboard feature controls security "before all" hook in "dashboard feature controls security"
[job] [logs] Default CI Group #19 / dashboard feature controls dashboard feature controls security "before all" hook in "dashboard feature controls security"
[job] [logs] Default CI Group #13 / Dev Tools feature controls security "before all" hook in "security"
[job] [logs] Default CI Group #13 / Dev Tools feature controls security "before all" hook in "security"
[job] [logs] Default CI Group #25 / discover preserve url "after all" hook for "remembers url after switching spaces"
[job] [logs] Default CI Group #25 / discover preserve url "after all" hook for "remembers url after switching spaces"
[job] [logs] Default CI Group #25 / discover preserve url "before all" hook for "goes back to last opened url"
[job] [logs] Default CI Group #25 / discover preserve url "before all" hook for "goes back to last opened url"
[job] [logs] Docker CI Group / endpoint "before all" hook in "endpoint"
[job] [logs] Docker CI Group / endpoint "before all" hook in "endpoint"
[job] [logs] Default CI Group #12 / graph app feature controls spaces "before all" hook in "spaces"
[job] [logs] Default CI Group #12 / graph app feature controls spaces "before all" hook in "spaces"
[job] [logs] Jest Tests #5 / has defaults for config
[job] [logs] Default Accessibility Tests / Kibana overview "after all" hook for "Kibana overview"
[job] [logs] Default Accessibility Tests / Kibana overview "after all" hook for "Kibana overview"
[job] [logs] Default Accessibility Tests / Kibana overview "before all" hook for "Kibana overview"
[job] [logs] Default Accessibility Tests / Kibana overview "before all" hook for "Kibana overview"
[job] [logs] Jest Tests #5 / Legacy config getGlobalConfig should return the global config
[job] [logs] Jest Tests #5 / Legacy config getGlobalConfig$ should return an observable for the global config
[job] [logs] Default CI Group #3 / lens app "after all" hook in "lens app"
[job] [logs] Default CI Group #16 / lens app "after all" hook in "lens app"
[job] [logs] Default CI Group #3 / lens app "after all" hook in "lens app"
[job] [logs] Default CI Group #16 / lens app "after all" hook in "lens app"
[job] [logs] Default CI Group #3 / lens app "before all" hook in "lens app"
[job] [logs] Default CI Group #16 / lens app "before all" hook in "lens app"
[job] [logs] Default CI Group #3 / lens app "before all" hook in "lens app"
[job] [logs] Default CI Group #16 / lens app "before all" hook in "lens app"
[job] [logs] Default CI Group #26 / machine learning anomaly detection single metric "after all" hook for "deletes the cloned job"
[job] [logs] Default CI Group #26 / machine learning anomaly detection single metric "after all" hook for "deletes the cloned job"
[job] [logs] Default CI Group #15 / machine learning permissions for user with full ML access with data loaded "after all" hook in "with data loaded"
[job] [logs] Default CI Group #15 / machine learning permissions for user with full ML access with data loaded "after all" hook in "with data loaded"
[job] [logs] Default CI Group #8 / machine learning settings calendar creation "after all" hook for "creates new calendar that applies to specific jobs"
[job] [logs] Default CI Group #8 / machine learning settings calendar creation "after all" hook for "creates new calendar that applies to specific jobs"
[job] [logs] OSS CI Group #9 / management "Create Index Pattern" wizard "before all" hook in ""Create Index Pattern" wizard"
[job] [logs] OSS CI Group #9 / management "Create Index Pattern" wizard "before all" hook in ""Create Index Pattern" wizard"
[job] [logs] OSS CI Group #8 / management index pattern filter "before all" hook for "should filter indexed fields"
[job] [logs] OSS CI Group #8 / management index pattern filter "before all" hook for "should filter indexed fields"
[job] [logs] Default CI Group #10 / maps app "after all" hook in "maps app"
[job] [logs] Default CI Group #22 / maps app "after all" hook in "maps app"
[job] [logs] Default CI Group #22 / maps app "after all" hook in "maps app"
[job] [logs] Default CI Group #9 / maps app "after all" hook in "maps app"
[job] [logs] Default CI Group #10 / maps app "after all" hook in "maps app"
[job] [logs] Default CI Group #9 / maps app "after all" hook in "maps app"
[job] [logs] Default CI Group #10 / maps app "before all" hook in "maps app"
[job] [logs] Default CI Group #22 / maps app "before all" hook in "maps app"
[job] [logs] Default CI Group #22 / maps app "before all" hook in "maps app"
[job] [logs] Default CI Group #9 / maps app "before all" hook in "maps app"
[job] [logs] Default CI Group #10 / maps app "before all" hook in "maps app"
[job] [logs] Default CI Group #9 / maps app "before all" hook in "maps app"
[job] [logs] Default CI Group #1 / Monitoring app feature controls security "after all" hook in "security"
[job] [logs] Default CI Group #1 / Monitoring app feature controls security "after all" hook in "security"
[job] [logs] Default CI Group #1 / Monitoring app feature controls security "before all" hook in "security"
[job] [logs] Default CI Group #1 / Monitoring app feature controls security "before all" hook in "security"
[job] [logs] Jest Tests #5 / response headers allows to configure "keep-alive" header
[job] [logs] OSS CI Group #7 / saved objects management saved objects inspect page allows to view the saved object
[job] [logs] OSS CI Group #7 / saved objects management saved objects inspect page allows to view the saved object
[job] [logs] Default CI Group #20 / saved objects security and spaces enabled _bulk_create user with no access within the default space "after all" hook for "should return 403 forbidden [hiddentype/any]"
[job] [logs] Default CI Group #20 / saved objects security and spaces enabled _bulk_create user with no access within the default space "after all" hook for "should return 403 forbidden [hiddentype/any]"
[job] [logs] Default CI Group #20 / saved objects security and spaces enabled _bulk_create user with no access within the default space "before all" hook for "should return 403 forbidden [isolatedtype/defaultspace-isolatedtype-id]"
[job] [logs] Default CI Group #20 / saved objects security and spaces enabled _bulk_create user with no access within the default space "before all" hook for "should return 403 forbidden [isolatedtype/defaultspace-isolatedtype-id]"
[job] [logs] Default CI Group #14 / saved objects tagging - functional tests table listing "after all" hook in "table listing"
[job] [logs] Default CI Group #14 / saved objects tagging - functional tests table listing "after all" hook in "table listing"
[job] [logs] Default CI Group #14 / saved objects tagging - functional tests table listing "before all" hook in "table listing"
[job] [logs] Default CI Group #14 / saved objects tagging - functional tests table listing "before all" hook in "table listing"
[job] [logs] Default CI Group #7 / security app Security Login Page "after all" hook for "displays message acknowledging logout"
[job] [logs] Default CI Group #7 / security app Security Login Page "after all" hook for "displays message acknowledging logout"
[job] [logs] Default CI Group #7 / security app Security Login Page "before all" hook for "can login"
[job] [logs] Default CI Group #7 / security app Security Login Page "before all" hook for "can login"
[job] [logs] Default CI Group #4 / Spaces app Copy Saved Objects to Space "after all" hook for "allows a dashboard to be copied to the marketing space, with circular references"
[job] [logs] Default CI Group #4 / Spaces app Copy Saved Objects to Space "after all" hook for "allows a dashboard to be copied to the marketing space, with circular references"
[job] [logs] Default CI Group #4 / Spaces app Copy Saved Objects to Space "before all" hook for "allows a dashboard to be copied to the marketing space, with all references"
[job] [logs] Default CI Group #4 / Spaces app Copy Saved Objects to Space "before all" hook for "allows a dashboard to be copied to the marketing space, with all references"
[job] [logs] Default CI Group #21 / transform feature controls security "after all" hook in "security"
[job] [logs] Default CI Group #21 / transform feature controls security "after all" hook in "security"
[job] [logs] Default CI Group #21 / transform feature controls security "before all" hook in "security"
[job] [logs] Default CI Group #21 / transform feature controls security "before all" hook in "security"
[job] [logs] Default CI Group #6 / Uptime app with real-world data "before all" hook in "with real-world data"
[job] [logs] Default CI Group #6 / Uptime app with real-world data "before all" hook in "with real-world data"
[job] [logs] OSS CI Group #11 / visualize app visualize ciGroup11 tag cloud chart "before all" hook for "should have inspector enabled"
[job] [logs] OSS CI Group #11 / visualize app visualize ciGroup11 tag cloud chart "before all" hook for "should have inspector enabled"

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/server-http-tools`	50	51	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`kbnUiSharedDeps-srcJs`	3.8MB	3.8MB	+20.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/server-http-tools`	53	54	+1

History

💔 Build #20101 failed 2e4fb3d
💔 Build #19481 failed 05dce08

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

Dosant · 2022-02-02T11:22:42Z

bfetch can be refactored to support http/2. Maybe we don't need it at all and can fall back to the plain HTTP call? HTTP/2 hasn't got limits on the number of parallel requests to a single domain.

@mshustov, I was thinking about this and thought that maybe there is not much benefit in having bfetch layer with HTTP/2, so added a configuration to not use it for search: #122244 so we could test the difference in cloud.

I thought that would be a good experiment because I saw h2 connection in cloud, but I completely missed that h2 connection is just between client and proxy 🤦‍♂️

TIL that's a lot more complicated

rudolf · 2023-01-24T14:24:39Z

Closing as stale PR

…183697) ## Summary Related to #7104 Extracted from #123748 `http2` headers are created via `Object.create(null)`, which causes issues with the way our `pick` method was implemented. PR fixes it.

restrry added 7 commits January 25, 2022 18:28

install http js client with http/2 support

7fe6ca9

add basic support for http/2 in Core Http service

dfd35dc

hack bfetch to be compatible with http2

3038db4

pick must work with an object without prototype

834fffa

use got client with http/2 support to send request to Kibana in FTR

d2f6cb0

I did not fix the base path proxy yet

5dcda95

prepare a few tests to be run in http/2 mode

05dce08

mshustov added Feature:http Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc labels Jan 25, 2022

mshustov commented Jan 25, 2022

View reviewed changes

restrry added 6 commits January 27, 2022 15:36

use hacked hapi version

faa3b40

do not run test/functional with secure connection

4ef6274

use http2 for the performance tests

2e4fb3d

fix tests

bb9f570

ignore node warning on unsupported headers in http2

03b7e52

Revert "use hacked hapi version"

5a1b2cf

This reverts commit faa3b40.

Dosant mentioned this pull request Feb 3, 2022

[Search] Benchmark bfetch impact in cloud with http2 #124538

Closed

mshustov mentioned this pull request Feb 8, 2022

Http2 support for the Kibana server #7104

Open

rudolf closed this Jan 24, 2023

This was referenced May 15, 2024

Add http2 support for Kibana server #183465

Merged

@kbn/std: fix pick method to work for objects without prototypes #183697

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[spacetime] Http/2 support #123748

[spacetime] Http/2 support #123748

mshustov commented Jan 25, 2022 •

edited

Loading

mshustov Jan 25, 2022

mshustov Jan 25, 2022

kibana-ci commented Jan 27, 2022 •

edited

Loading

API count

Dosant commented Feb 2, 2022 •

edited

Loading

rudolf commented Jan 24, 2023

[spacetime] Http/2 support #123748

[spacetime] Http/2 support #123748

Conversation

mshustov commented Jan 25, 2022 • edited Loading

Summary

Why

TODO

Performance impact

on the loading time

Known problems

mshustov Jan 25, 2022

Choose a reason for hiding this comment

mshustov Jan 25, 2022

Choose a reason for hiding this comment

kibana-ci commented Jan 27, 2022 • edited Loading

💔 Build Failed

Failed CI Steps

Test Failures

Metrics [docs]

Public APIs missing comments

Page load bundle

API count

History

Dosant commented Feb 2, 2022 • edited Loading

rudolf commented Jan 24, 2023

mshustov commented Jan 25, 2022 •

edited

Loading

kibana-ci commented Jan 27, 2022 •

edited

Loading

Dosant commented Feb 2, 2022 •

edited

Loading