Commits on Feb 9, 2023

1. [Fleet] Run agent policy schema in batches during fleet setup + add `… …

   …xpack.fleet.setup.agentPolicySchemaUpgradeBatchSize` config (elastic#150688)
   
   ## Summary
   
   Closes elastic#150538
   
   As part of the Fleet plugin setup, we check to see if any agent policies
   have an out of date `schema_version` and upgrade them. We encountered an
   error when this upgrade happens on a large number of agent policies as
   we attempted the upgrade in one large batch.
   
   This pull request performs the schema upgrade in batches of 100 by
   default and also adds the config value
   `xpack.fleet.setup.agentPolicySchemaUpgradeBatchSize` to make the batch
   size configurable.
   
   I have also added more debug logging to show progress, and reduced the
   response payload of one of our requests which was very large.
   
   ### Dev testing
   
   To test this you need an environemnt with lots of agent policies (> 2k)
   where `schema_version`
   is not set. To create an environment with a large number of agent
   policies I have added a new param to the agent creation script, I ran:
   
   ```
   cd x-pack/plugins/fleet
   node scripts/create_agents --count 20  --kibana http://127.0.0.1:5601/mark --status online --delete --batches 3000 --concurrentBatches 100
   ```
   
   To generate 3000 agent policies each with 20 agents in.
   
   I then modified the agent policies so that they require an upgrade, as
   `system_indices_superuser` run:
   
   ```
   POST /.kibana/_update_by_query
   {
     "query": {
       "bool": {
         "filter": [
           {
             "term": {
               "type": "ingest-agent-policies"
             }
           }
         ]
       }
     },
     "script": {
       "source": "ctx._source['ingest-agent-policies'].remove('schema_version')",
       "lang": "painless"
     }
   }
   ```
   
   restarting kibana will run the setup and in batches.
   
   (cherry picked from commit 6e06452)
   
   # Conflicts:
   #	x-pack/plugins/fleet/scripts/create_agents/create_agents.ts

   

   hop-dev committed Feb 9, 2023
   Configuration menu

   • View commit details
   • Copy full SHA for c61fb88
   • Browse repository at this point

   Copy the full SHA

   c61fb88 View commit details

   Browse the repository at this point in the history