Allow Kibana to setup the stack

[ruflin]

With the introduction of Fleet and Solutions more and more assets have to be loaded into Elasticsearch and Kibana on startup / upgrade to prepare the stack. Not only saved objects need to be loaded, but also assets like Elasticsearch ingest pipelines, Elasticsearch index templates, ILM policy and others. At the moment, this setup step requires a user to login or an API call with sufficient permissions to trigger it. The reason is that the kibana_system user does not have sufficient permissions. From a usability perspective this is not great, especially after upgrades.

This issue is to discuss how Kibana could become capable of setting up the stack.

Related issues / problems

The following is a list of issue where we hit the above issue. This will be extended over time and reference issues will be added.

• Initial setup of fleet and packages: [Fleet] Configure Fleet packages and integrations through config file #88956
• Bootstrapping of fleet-server indices / templates: Bootstrapping of Elastic indices for Fleet / Fleet-Server, Phase 1 fleet-server#51
• Setup required packages like system and endpoint on Kibana startup
• Upgrade endpoint package when Kibana is upgraded
• Upgrade Elastic Agents when Kibana is upgraded [Fleet] Upgrade Elastic Agent in the background #75651
• Upgrade Fleet's final pipeline when Kibana is upgraded [Fleet] Support migration for fleet final pipeline #106415

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[tsg]

Thank you @ruflin for posting this ticket! On the Security side, we need a user with sufficient privileges to visit the "Manage Rules" page in order to bootstrap the indices that are used to store Alerts and to enable the default promotion rule. This is needed once for each space.

One thing to note is that as we have more and more places in Kibana that require user action, the effects on the getting started experience are compounded. If you land on a page that requires bootstrapping to work, we need to show an intermediary screen like these:

When setting up a new cluster on Cloud, selecting Security as the use case, there are 4 or 5 such screens before you get ingestion going:

• After Kibana boots up, the user is sent to the Security Overview page. A banner prompts them to install an Endpoint.
• Clicking the banner gets you to enable the Endpoint integration. After enabling that, you are sent to install an agent.
• To install an agent, you need Fleet bootstrapped, so you are sent the Fleet page
• After installing the agent and getting data in, you still need to visit the Manage Rules page to setup Alerting.

While each of these steps is only a click or two, the fact that the user has to go through multiple pages and apps to get things started seems less than ideal, and will likely get worse as we add more functionality. CC @VijayDoshi as this might be related to the Getting Started WG.

[alvarolobato]

@sqren this is the same problem you had for the background job in APM, when working on the service map right? was there anything else?

[sorenlouv]

Thanks for opening this important discussion @ruflin!

this is the same problem you had for the background job in APM, when working on the service map right? was there anything else?

Yes, exactly the same problem. We wanted to have a background job that transformed the raw APM data into a format tailor-made for the Service Map to improve end-user performance.
This was not easily doable since The kibana_system didn't have access to the APM indices. Even if we could give it access to apm-* (which the Security team was not super happy about) it wouldn't work.
Since the APM indicies are configurable we'd have to give the kibana_system user access to all indices to ensure it could read APM data.
This last part should be greatly simplified with Data Streams where the indicies are hard coded (non-configurable).

On the Security side, we need a user with sufficient privileges to visit the "Manage Rules" page in order to bootstrap the indices that are used to store Alerts and to enable the default promotion rule. This is needed once for each space.

This was also the direction we were going in. But due to all the issues that came with it we stuck with a less performant runtime solution

[andrewvc]

On the Synthetics side we need this as well. Our needs:

1. Synthetics package versioned pinned to an exact version of the stack (ideally just shipped with Kibana, no registry needed)
2. No upgrade/downgrade package functionality visible. There's no analogous feature to 'upgrading the synthetics package' in competing products that I've ever heard of.

So, what this means is the package should ideally (but not necessarily required) be shipped with the stack, and upgraded / downgraded on boot. UI-wise any fleet UI features related to package version would be omitted (or grayed out).

[axw]

APM would like to have the same experience as described above.

Apart from enabling better UX by removing any need to manage package versions independently, this will also enable simpler (more maintainable) UI code by allowing it to make more assumptions about the data streams with which it's working.

The Elasticsearch team are exploring the ability to define runtime fields at the data stream level. If we automatically upgrade the package, ensuring templates, pipelines, etc. are kept in sync with the stack version, then the UI will be able to assume the presence of runtime fields and avoid complex conditional logic.

[dikshachauhan-qasource]

Hi @ruflin

Could you please be able to provide us info on AC of this ticket. Majorly, about how can we validate the merges whenever available.

Thanks
QAS

[joshdover]

@dikshachauhan-qasource This was a meta issue to track several related changes that we've been testing already. I don't expect any explicit action needed from the QA team on this.

Separately, I think we should now close this issue as all major pieces have been handled. The last remaining issue for upgrading Agents is somewhat tangential to the other items and this feature is not yet GA. Let's continue the discussion in #75651