[8.x](backport #6619) enhancement(5423): added logic to replaces scheduler with long-wait scheduler in case of exceeded unauth response limit

[mergify]

• Enhancement

What does this PR do?

Removes the forced unenroll from fleet gateway. Adds logic in the fleet gateway to switch out the scheduler used for checkins. If the unauthorized response limit is exceeded, a the scheduler is replaced with one that has a long wait duration. When the gateway receives a successful response, it switches back to using the regular scheduler with the shorter wait duration.

Why is it important?

Currently the agent unenrolls after 7 unauthorized error responses. This can causes problems in disaster recovery scenarios where users may have to manually intervene.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

None

How to test this PR locally

• Create ESS deployment
• Build the agent locally
• Enroll the agent
• In dev tools find the access token and delete it

GET /.security-7/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "name": "AGENT ID"
          }
        }
      ]
    }
  }
}

DELETE /_security/api_key
{
  "ids": ["KEY ID"]
}

• Follow the agent logs sudo elastic-agent logs -f
• After a while you will see retrieved an invalid api key error '10' times. will use long scheduler error message in the logs

Due to the backoff algorithm used, this test can take a long time. In order to see immediate results comment out the following code block

			if !bo.Wait() {
				if ctx.Err() != nil {
					// if the context is cancelled, break out of the loop
					break
				}

				// This should not really happen, but just in-case this error is used to show that
				// something strange occurred and we want to log it and report it.
				err := errors.New(
					"checkin retry loop was stopped",
					errors.TypeNetwork,
					errors.M(errors.MetaKeyURI, f.client.URI()),
				)

				f.log.Error(err)
				f.errCh <- err
				return nil, err
			}

Related issues

• Closes Remove automatic unenrollment after 7 Fleet authentication failures #5428
• Relates Gradually ramp up to wait duration when agent runs into more than 7 authentication errors #6760

---

This is an automatic backport of pull request #6619 done by [Mergify](https://mergify.com).

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
71.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube