[8.6] [Fleet] Run agent policy schema in batches during fleet setup + add xpack.fleet.setup.agentPolicySchemaUpgradeBatchSize config (#150688)

[hop-dev]

Backport

This will backport the following commits from main to 8.6:

• [Fleet] Run agent policy schema in batches during fleet setup + add xpack.fleet.setup.agentPolicySchemaUpgradeBatchSize config (#150688)

Questions ?

Please refer to the Backport tool documentation

\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sqren/backport)\n\n\n\nCo-authored-by: Mark Hopkin "}},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com//pull/150688","number":150688,"mergeCommit":{"message":"[Fleet] Run agent policy schema in batches during fleet setup + add `xpack.fleet.setup.agentPolicySchemaUpgradeBatchSize` config (#150688)\n\n## Summary\r\n\r\nCloses #150538 \r\n\r\nAs part of the Fleet plugin setup, we check to see if any agent policies\r\nhave an out of date `schema_version` and upgrade them. We encountered an\r\nerror when this upgrade happens on a large number of agent policies as\r\nwe attempted the upgrade in one large batch.\r\n\r\nThis pull request performs the schema upgrade in batches of 100 by\r\ndefault and also adds the config value\r\n`xpack.fleet.setup.agentPolicySchemaUpgradeBatchSize` to make the batch\r\nsize configurable.\r\n\r\nI have also added more debug logging to show progress, and reduced the\r\nresponse payload of one of our requests which was very large.\r\n\r\n### Dev testing\r\n\r\nTo test this you need an environemnt with lots of agent policies (> 2k)\r\nwhere `schema_version`\r\nis not set. To create an environment with a large number of agent\r\npolicies I have added a new param to the agent creation script, I ran:\r\n\r\n```\r\ncd x-pack/plugins/fleet\r\nnode scripts/create_agents --count 20 --kibana http://127.0.0.1:5601/mark --status online --delete --batches 3000 --concurrentBatches 100\r\n```\r\n\r\nTo generate 3000 agent policies each with 20 agents in.\r\n\r\nI then modified the agent policies so that they require an upgrade, as\r\n`system_indices_superuser` run:\r\n\r\n```\r\nPOST /.kibana/_update_by_query\r\n{\r\n \"query\": {\r\n \"bool\": {\r\n \"filter\": [\r\n {\r\n \"term\": {\r\n \"type\": \"ingest-agent-policies\"\r\n }\r\n }\r\n ]\r\n }\r\n },\r\n \"script\": {\r\n \"source\": \"ctx._source['ingest-agent-policies'].remove('schema_version')\",\r\n \"lang\": \"painless\"\r\n }\r\n}\r\n```\r\n\r\nrestarting kibana will run the setup and in batches.","sha":"6e06452aac11ed22efa923284fbb9ad4da1f7ce1"}}]}] BACKPORT-->

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: c61fb88

Metrics [docs]

Unknown metric groups

ESLint disabled in files

id	before	after	diff
osquery	1	2	+1

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
fleet	61	67	+6
osquery	108	113	+5
securitySolution	441	447	+6
total			+19

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
fleet	69	75	+6
osquery	109	115	+6
securitySolution	518	524	+6
total			+20

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream