Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the optimizer from builds #7322

Closed
2 of 9 tasks
epixa opened this issue May 28, 2016 · 9 comments · Fixed by #73154
Closed
2 of 9 tasks

Remove the optimizer from builds #7322

epixa opened this issue May 28, 2016 · 9 comments · Fixed by #73154
Assignees
Labels
high hanging fruit release_note:enhancement ReleaseStatus Item of high enough importance that it should be called out in release status meetings Team:Operations Team label for Operations Team

Comments

@epixa
Copy link
Contributor

epixa commented May 28, 2016

The optimizer runs in kibana releases whenever you install a plugin and whenever you start kibana after removing a plugin. This shouldn't be necessary or even possible.

The bundles that get created at build time and shipped with the core release should be immutable, and all plugins should need to create their own bundles during their own build process. Kibana should not in any way attempt to wrangle the actual plugin code itself, and plugins should not in any way be concerned with Kibana doing so.

This should mean that the optimizer does not get shipped with builds and is only used in development. That will greatly speed up plugin install/removal, will improve reliability of plugin installs and the integrity of the core kibana build, remove the need for certain permission wrangling that occurs now for package installs, and will allow us to ship Kibana with defaults for low max memory usage.

This should also free us to explore changes to the entire dev build system to make it faster and easier to use since we won't be burdened by production requirements of the optimizer. Ultimately, it will allow us to treat our tooling around the dev process as simply a part of the project rather than as part of the product itself.

Blockers:

Tasks:

  • Disable the legacy optimizer in development
  • Disable legacy optimizer in production
  • Remove clustering mechanism
  • Remove src/optimizer and related dependencies
  • Remove karma & related CI tasks(Tim Roes)

Closes #19633
Closes #19678
Related to #57406

@epixa
Copy link
Contributor Author

epixa commented May 28, 2016

/cc @tylersmalley

@spinus
Copy link

spinus commented Jul 11, 2017

Would love to have that.

@epixa epixa changed the title Kibana builds should not have any rebuilding capabilities Remove the optimizer from builds May 3, 2018
@epixa
Copy link
Contributor Author

epixa commented May 3, 2018

I removed the breaking label because this isn't necessarily a breaking change.

@epixa
Copy link
Contributor Author

epixa commented May 7, 2018

Once this is done, we can just close #5071 if that's still open.

@bradvido
Copy link

Will this also fix #19678 ?

@mistic
Copy link
Member

mistic commented Jul 13, 2018

I have some updates on this matter. Over the past days I spent some time investigating ways to kill/drastically reduce the use of optimizer on Kibana. I have also spent some time in a POC you can found here mistic/webpack-dll-spike-app.

Removing the optimizer at this stage, if possible, will be very difficult as we in the current architecture we hardly rely on it. However I (and other folks on the @elastic/kibana-operations ) believe that could be possible to follow another path, that instead of remove the optimizer will help us to drastically reduce the use of it and move us in a step further into the goal of removing it! The main ideas behind this possible solution rely on the webpack DLL (I will narrow the description to a production production situation):

1st Step - Kibana Side - Building Kibana

  • Add a DLL bundler to the optimizer with 2 main pieces: a compiler and a loader.
  • In the first stage (during our CI pipeline) the compiler runs and builds the needed DLLs for Kibana (vendors_client, vendors_server, kibana_client, kibana_server).
  • The loader kicks in and generate the main entry bundles referencing the generated Dlls and generating a first main entry bundle (this should be super fast!).
  • Kibana gets released with the generated assets, the dlls and the main entry bundle, as long as with the bundler itself.
  • When started in prod Kibana will always referencing all the Dlls it haves and regenerating the main entry bundle, then start.

2nd Step - Plugin Side - Building a Plugin

  • For the plugins the flow is the same. The plugin template ships the DLLBundler and downloads the Kibana DLLs for a specific version.
  • Runs the compiler to generate the plugin DLLs, and referencing the Kibana DLLs.
  • The plugin gets distributed as form as a DLL.

3rd Step - Kibana Side - Importing a Plugin

  • In order to finally importing a plugin, it is just need to run a plugin-installer to download the plugin DLL and put it in the kibana Dlls folder.
  • Re-run the Bundler Loader to regenerate the main entry files
  • Restart kibana server and we are done! (another approach could be follow like for example spin up a second kibana server and kills the older when the new is started)

We believe that this could be a possible solution for the problem, and it will also allow us moving in the direction of being able to remove the optimizer from production. You can found some of the concepts in the above mentioned POC. As the first step we decide trying to implement a vendors dll, only for the client side, and then re-evaluate! A new issue #20749 was created to accomplish it.

@joshdover
Copy link
Contributor

joshdover commented Jul 1, 2020

Remaining plugins that still rely on the optimizer in production:

We also have the "tests_bundle" plugin which is used for building a bundle of tests to be executed inside a real browser using Karma. I believe we should be able to remove the optimizer from production builds before we have a replacement mechanism for this. However, it may be simpler to migrate all of this first and just completely remove the legacy optimizer rather than attempting to have special prod-only behavior.

Virtually all the remaining Karma tests are in the kibana plugin. We have some in ui/public but these should be safe to delete completely. The Kibana App team has an issue for converting these tests over to jest (#57813). @timroes if we get to a point soon where this is the last blocker to removing the legacy optimizer, how comfortable would we be with disabling these tests until your team has time to migrate them to jest? Do we see breakages in these visualizations often?

@timroes
Copy link
Contributor

timroes commented Jul 13, 2020

Just to leave this in written state here: We're working on migrating the Karma tests really soon all over to other test frameworks. If there are still remaining tests left after 7.10, we're fine disabling/removing them until we can migrate them to a different test framework.

@stacey-gammon stacey-gammon added the ReleaseStatus Item of high enough importance that it should be called out in release status meetings label Sep 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high hanging fruit release_note:enhancement ReleaseStatus Item of high enough importance that it should be called out in release status meetings Team:Operations Team label for Operations Team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

9 participants