[Fleet] Only enable secrets storage if all Fleet Servers are at a minimum supported version

[juliaElastic]

Part of #154715

For backwards compatibility, even after the secrets storage work is GA, we want to ensure that secrets storage is turned on when all the Fleet Servers are on a minimum supported version (e.g. 8.9 or 8.10).

• Add a check on package policy creation that Fleet Servers version meet the requirement. If so, enable secret storage.
  • Package policy update should continue to store secrets as they were created. Or does it make sense to move from insecured to secured secrets on package policy update once Fleet Servers support it?
• (Optional): Add a UI warning when there is at least one Fleet Server on lower version, and the package has secrets, that the secrets are not going to be stored securely.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[hop-dev]

I believe this could be as simple as replacing the current feature flag check with the fleet server version check.

We should prominently log that we are not storing secrets securely, but probably not every time as that could get quite noisy?

@juliaElastic One area that may need scoping here, should we inform the user that their secrets are not being stored securely because of their fleet server version? We could have a dismissible banner on the agents page informing them. I guess it may not even have to mention secrets, just that keeping your fleet server up to date means you get the latest features. Maybe one for @nimarezainia

[juliaElastic]

I think it would be best to always log the result of the Fleet Server check in case we get SDHs and want to investigate whether the secrets logic has a bug or it was disabled because of old Fleet Servers, hopefully not going to happen often.

I think for the UI we should add a banner that can be dismissed. I would let the exact wording to @nimarezainia, at least mentioning that using the latest Fleet Servers is recommended to get the latest security features.

[hop-dev]

One thought I have had is that we could simplify this by saying that once secrets are "on" they can't be turned off. I.e once the user has met the fleet server requirement, secrets are enabled and even if they add an old fleet serve from that point we will still continue to use them.

This way we don't have to deal with the scenario where a policy has secrets, but then they add an old fleet server, we would not be able to revert to not usng secrets in that scenario because we cannot read the values from the secrets index.

We would need a place to store that secrets are enabled, maybe a new saved object (that is hidden and not exportable)? Then the check would be:

• check for the existence of the secrets enabled saved object
• if it exists then secrets are enabled (maybe cache this result so we dont have to do the saved objects query every time)
• if it doesn't exist then get all fleet servers
• if one fleet server is below 8.9 then return false
• if all fleet servers are 8.9 or above, create the secrets enabled saved object and return true

[hop-dev]

If a user does not have a fleet server I think secrets should be disabled. This is because if they proceed to add a fleet server of a lower version, their policy breaks. Whereas disabling secrets and then re-enabling them is a non breaking change.

[hop-dev]

@juliaElastic @joshdover do you agree with this approach? Just checking no conversations have been had outisde of this issue.

One thought I have had is that we could simplify this by saying that once secrets are "on" they can't be turned off. I.e once the user has met the fleet server requirement, secrets are enabled and even if they add an old fleet serve from that point we will still continue to use them.

This way we don't have to deal with the scenario where a policy has secrets, but then they add an old fleet server, we would not be able to revert to not usng secrets in that scenario because we cannot read the values from the secrets index.

We would need a place to store that secrets are enabled, maybe a new saved object (that is hidden and not exportable)? Then the check would be:

• check for the existence of the secrets enabled saved object
• if it exists then secrets are enabled (maybe cache this result so we dont have to do the saved objects query every time)
• if it doesn't exist then get all fleet servers
• if one fleet server is below 8.9 then return false
• if all fleet servers are 8.9 or above, create the secrets enabled saved object and return true

[joshdover]

Overall plan makes sense to me, a couple thoughts:

If a user does not have a fleet server I think secrets should be disabled. This is because if they proceed to add a fleet server of a lower version, their policy breaks. Whereas disabling secrets and then re-enabling them is a non breaking change.

I wish we could avoid this, but I see your point. Another option could be to show a warning in the UI if an Agent < 8.10 w/ Fleet Server policy is enrolled, prompting the user to upgrade it. My assumption being that even if output secrets aren't working that the Fleet Server agent should still be able to receive upgrade actions.

We also need to account for the Serverless case where there will be no Fleet Server agents. In this case secrets should always be enabled - I think we can probably drive this from one of the kibana.yml configs.

maybe a new saved object

SO makes sense, but let's see if we have any existing settings SO we can use. I'd rather not have a specific SO type just for a single boolean flag. If we don't have one, let's just make the type name rather generic so we can use it for other things in the future.

[juliaElastic]

I agree with this (the condition with 8.10 if we make it), we could use the existing ingest_manager_settings SO type that only has another boolean now:

  "ingest_manager_settings": {
            "prerelease_integrations_enabled": false
          },