[Fleet] Implement state machine behavior for package install

[criamico]

Phase 1 of #169147

Implement a state machine like behavior for the package install:

• Refactor the existing steps as steps that can be executed independently. Main steps:
  • kibana_assets
  • ilm_policies
  • index_templates_and_pipelines
  • transforms
  • ml_models
• Keep track of the current step and save it, so that the next install can be restarted from there. To do that we need to persist the status in the installation saved object (exact format TBD)
• For each state transition:
  • Implement and check initial conditions that need to be verified prior to execute the state
  • Implement retries and recovery steps to execute when the state failed
  • Expose the current state and errors, if present, in the install endpoint response (POST api/fleet/epm/packages/<pkgName>/<pkgVersion>)
  • Introduce a flag that allows to do the whole reinstall from scratch (we could reuse force for that)
• Add tests covering the new functionality

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

Refactor the existing steps as steps that can be executed independently

@criamico I do not think this is needed until we have independent API no?

[criamico]

I do not think this is needed until we have independent API no?

@nchaulet I didn't intend that those functions need to be exposed but just tweaked the strict necessary to make them part of a state machine. Some work will be needed in those functions, although minimal, and I wanted to highlight that. But we can rephrase it to make it clearer.

[criamico]

@nchaulet and I had a discussion about how to implement the state machine behavior. In the RFC we didn't get to discuss this aspect.

We considered using a library like XState. It is already used inside Kibana (see https://github.com/elastic/kibana/tree/main/x-pack/plugins/observability_solution/infra/docs/state_machines), but it's actually pretty complex and we agreed that we should strive to to keep the implementation simple. Plus we probably don't need all these functionalities for our case.

From our discussion it came out that we need the following:

• Provide the states and the functions to execute on each state in a way that's simple to write and maintain
• Persist the current install state (similar to what we already do with install_failed state)
• Have a way to restart the steps from any state when the install failed
• Keep the state machine code decoupled from the actual lifecycle functionality, so it can be fully tested
• Define states based on the existing functions in

  kibana/x-pack/plugins/fleet/server/services/epm/packages/_install_package.ts

  Line 64 in 35d037b

  export async function _installPackage({

Based on these considerations I created a branch with a small POC. All it does is to create a state machine based on a data structure - note that for now the functions are empty as it's just an example. The unit tests show how it should work with a generic data structure.

@kpollich I'd also like to have your take on this before going too far with it.

[kpollich]

XState is very good but I worry moving to it would be too large a refactor here, so I agree with going a different direction.

I'll take a look at the draft PR and let you know if I have any other thoughts. Thanks for checking in here.

[criamico]

PR was merged. I added the tasks not covered by #178657 in #175597

[criamico]

PR to add more APM spans: #180355