New build toolchain

[tylersmalley]

Kibana has grown over the years, but it's been a lot of work to scale to what we have today. A lot of this work was due to the fact we used Webpack as our primary build system and it had to include the code from all plugins within Kibana. These scaling issues were primarily related to the performance (time) and memory required. Now that we have the new platform, that is no longer a blocker.

We also transform all of our Javascript, so I think it's appropriate that we utilize tooling more suited towards compiled languages. I have looked into many tools but the one which stood out was Bazel. Bazel is a proven build tool which has invested a lot of effort into Node/Javascript/Typescript specific rules (bazelbuild/rules_nodejs).

Features of Bazel

• Incremental - Rebuild time is proportional to what you changed
• Deterministic - We can cache buildresults based on their inputs
• Hermetic - allows features like remote execution, parallelization
• Composable - Bazel plugins are like Unix pipes, allows novel chaining
• Industrial grade - Based off the internal build tool at Google, Blaze.

With Bazel, we can take advantage of Remote caching. CI would be the only thing which wrote to the cache, but developers would have re-only access to reduce the amount of work needed by their local machine.

I have a small POC which is a work-in-progress here.

One of the idiomatic differences with Bazel than in the current Javascript world is that actions like Typescript transformations and minification are split up. This allow for Bazel to divvy up the work and to improve caching. For example, in Bazel your Typescript code with be transformed, then that transformed code would be the input into your Jest test. Compared that to using the jest-typescript plugin which uses Babel to transform the code used in Jest. The result is less dependencies and a much clearer and debugable code path.

Closes #62266

[tylersmalley]

This is a work in progress, so please feel free to add to the discussion.

[weltenwort]

Really looking forward to this! 👍

As far as globals go, we could create a bundle that just exposes globals via the expose-loader.

[c4milo]

Any update on this? this is a big deal, every time we change xpack.security.enabled, node consumes 100% CPU and takes a looooonng time to bind the service the port.

[epixa]

@c4milo We're still working on this, but there's still a huge effort ahead of us. We can't undo the webpack stuff in production builds until we have a fundamentally new approach in place for how plugins interact with each other, so that hinges heavily on our new platform project.

#12501 is tracking the initial effort around a build system in the new platform, if you are interested in the first step in the process.

I can't say for certain how this rollout will look, but our expectation now is that once the initial platform lands in the product, webpack will still be a part of production builds, but the optimization time and memory requirements will lessen with every release. The time/memory requirements will continue to diminish until we remove webpack entirely in 7.0.

[c4milo]

@epixa Thanks for the update! is there any workaround we can use when building kibana container images so that bundling and optimization doesn't happen in runtime? We are currently shipping our images with xpack.security.enabled: true so that's not likely to change in production.

[epixa]

@c4milo I'm not familiar with a specific workaround, but I'm curious - if you're using x-pack, why change the enable flag of security at all?

[c4milo]

@epixa we were having a redirect loop with Kibana and were trying to troubleshoot the issue. Runtime bundling and optimization got in the way in our process to narrow down the problem. That said, we are still having issues with Kibana plugins being RED state for some unknown reason.

[epixa]

Ah, I see. We have had some issues with x-pack security causing infinite redirects when Elasticsearch is inaccessible of the license is no longer valid. They should be fixed in the upcoming 5.6.0 release.

I highly recommend seeking help from the forums for unexpected plugin behaviors in general. We reserve github for bug reports and feature requests, so sometimes comments seeking guidance can go unanswered for awhile, whereas people read the forums with the intention of answering questions and helping people troubleshoot every day.

[jinmu03]

@mistic Can you break this down to multiple phases and provide rough release date for each phase?

[tylersmalley]

We're closing in favor of #69706 where we are specifically tracking migrating to Bazel.