[Fleet] Secrets are visible after saving package policy

[kpollich]

After creating a package policy for the AWS S3 policy template, secret values can be retrieved via the replace button in the secret field UI.

The secret values are also stored in plaintext on the resulting package policy saved object - this is the root of the issue.

Steps to reproduce

1. Create package policy for AWS S3 integration (policy template)
2. Fill in secrets for access key id, secret access key, session token
3. Save policy
4. Edit newly-created policy
5. Observe that clicking "replace secret" button on each secret variable allows user to see the secret in plaintext
6. Observe that the package policy saved object contains a value for each of these secrets as well, e.g.

"vars": {
  "shared_credential_file": {
    "type": "text"
  },
  "credential_profile_name": {
    "type": "text"
  },
  "access_key_id": {
    "type": "password",
    "value": "123123123"
  },
  "secret_access_key": {
    "type": "password",
    "value": "123123123"
  },
  "session_token": {
    "type": "password",
    "value": "123123123"
  },
  "role_arn": {
    "type": "text"
  },
  "default_region": {
    "value": "",
    "type": "text"
  },
  "proxy_url": {
    "type": "text"
  }
}

Screen recording

Screen.Recording.2024-03-05.at.8.53.06.AM.mov

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

This is affecting all secrets, not just package level secrets.

[kpollich]

I received this as a bug report in Slack, but I think what might be going on is the user doesn't have Fleet Server running, so secrets storage is force-disabled by Fleet.

I wonder if we should loosen this behavior, as a user could boot up Fleet Server at any time to begin decoding secrets. @elastic/fleet any thoughts?

[kpollich]

Also, we probably shouldn't show the secrets editor UI components when secrets storage is disabled.

[kpollich]

I took some time to test secrets across multiple stack upgrades to simulate a long running deployment to see if I could shake any issues loose related to this. See my steps + observations below:

1. Created a cluster on 8.9.2
2. Created a package policy for AWS S3 on the aws-2.9.1 integration

3. Upgrade cluster to 8.10.4. Note new AWS version is available

4. Upgrade integration + policies. Note no secrets in use quite yet

5. Upgrade cluster to 8.11.2. Observe new version of AWS is available again

6. Upgrade without also upgrading policies this time, just to verify what the state of once secrets are supported

7. Observe Upgrade button is available on integration policies table

8. Observe secrets onboarding callout is displayed

9. Can view secrets via "replace" button in UI

10. Save policy, observe that secrets can be still be revealed

11. Observe secret values still exist in plaintext on the policy saved object, e.g.

GET kbn:/api/fleet/package_policies/bc0eec6d-3067-4695-beb8-5d4ffe9a3ab4

"vars": {
    "shared_credential_file": {
      "type": "text"
    },
    "credential_profile_name": {
      "type": "text"
    },
    "access_key_id": {
      "type": "text",
      "value": "123_a"
    },
    "secret_access_key": {
      "type": "text",
      "value": "456_b"
    },
    "session_token": {
      "type": "text",
      "value": "789_c"
    },
    "role_arn": {
      "type": "text"
    },
    "default_region": {
      "value": "",
      "type": "text"
    },
    "proxy_url": {
      "type": "text"
    }
  },

---

One potential reason for the issue: secrets storage isn't actually enabled on my cluster because there's still an 8.9.2 Fleet Server enrolled, it's just offline:

I don't think our logic for enabling secrets storage accounts for offline Fleet Servers, e.g.

kibana/x-pack/plugins/fleet/server/services/secrets.ts

Lines 601 to 651 in 9413042

	export async function isSecretStorageEnabled(
	esClient: ElasticsearchClient,
	soClient: SavedObjectsClientContract
	): Promise<boolean> {
	const logger = appContextService.getLogger();
	// first check if the feature flag is enabled, if not secrets are disabled
	const { secretsStorage: secretsStorageEnabled } = appContextService.getExperimentalFeatures();
	if (!secretsStorageEnabled) {
	logger.debug('Secrets storage is disabled by feature flag');
	return false;
	}
	// if serverless then secrets will always be supported
	const isFleetServerStandalone =
	appContextService.getConfig()?.internal?.fleetServerStandalone ?? false;
	if (isFleetServerStandalone) {
	logger.trace('Secrets storage is enabled as fleet server is standalone');
	return true;
	}
	// now check the flag in settings to see if the fleet server requirement has already been met
	// once the requirement has been met, secrets are always on
	const settings = await settingsService.getSettingsOrUndefined(soClient);
	if (settings && settings.secret_storage_requirements_met) {
	logger.debug('Secrets storage requirements already met, turned on in settings');
	return true;
	}
	// otherwise check if we have the minimum fleet server version and enable secrets if so
	if (
	await allFleetServerVersionsAreAtLeast(esClient, soClient, SECRETS_MINIMUM_FLEET_SERVER_VERSION)
	) {
	logger.debug('Enabling secrets storage as minimum fleet server version has been met');
	try {
	await settingsService.saveSettings(soClient, {
	secret_storage_requirements_met: true,
	});
	} catch (err) {
	// we can suppress this error as it will be retried on the next function call
	logger.warn(`Failed to save settings after enabling secrets storage: ${err.message}`);
	}
	return true;
	}
	logger.info('Secrets storage is disabled as minimum fleet server version has not been met');
	return false;
	}

Pulling the cluster logs for my Kibana instance I can see that secrets storage is indeed disabled:

So, this is actually working as expected today - we won't store secrets unless all detected Fleet Servers are on 8.10.0 or greater. We should probably make two improvements in the short term:

1. Update the UI so we don't display secrets elements when secrets storage is disabled on the backend (in progress)
2. Fix the Fleet Server check to assert that as long as one Fleet Server exists above 8.10.0 then secrets can be enabled

I'm working on a PR to fix the UI elements so we don't show various secrets form elements when secrets storage is actually not enabled on the backend by this implicit check.

Longer term, should we even keep the Fleet Server version -> secrets enabled/disabled gate in place? It seems to just add confusion and implicitness. What are the risks of removing it?

[jillguyonnet]

Thanks for the detailed analysis report! I have a couple of questions around the Fleet Server related criteria to enable secrets.

Fix the Fleet Server check to assert that as long as one Fleet Server exists above 8.10.0 then secrets can be enabled

Longer term, should we even keep the Fleet Server version -> secrets enabled/disabled gate in place? It seems to just add confusion and implicitness. What are the risks of removing it?

If there is one Fleet Server on a version lower than 8.10.0, could there be a risk of breaking associated policies if secrets are enabled? AFAIK there is no granularity to secrets enabling.

I received this as a bug report in Slack, but I think what might be going on is the user doesn't have Fleet Server running, so secrets storage is force-disabled by Fleet.

I wonder if we should loosen this behavior, as a user could boot up Fleet Server at any time to begin decoding secrets. @elastic/fleet any thoughts?

Pretty much same question: if the user then adds a Fleet Server on a version lower than 8.10.0, could it break their policies?

Linking the original issue for reference: #157456

[juliaElastic]

Pretty much same question: if the user then adds a Fleet Server on a version lower than 8.10.0, could it break their policies?

I think we might want to optimise for the most common use case: if a user start using a newest version of kibana, it is likely that they will use a newest version of fleet-server with it.
So I would lean to enable secrets if there are no fleet-servers, and add a warning that enrolling an older fleet-server would not work - possibly with a way to disable secrets with a button / automatically if someone really wants to use an old fleet-server version.