Migrate status & stats APIs to KP + remove legacy status lib

[joshdover]

Summary

Related #41983

This migrates the /api/status and /api/stats endpoints to the Kibana Platform. It also deletes all of the src/legacy/server/status module and adapts any leftover code that was still using it. The updated status endpoint will now report statuses from all Kibana Platform plugins & core services.

Some discussion points:

• While these two endpoints are very similar, /api/stats relies on UsageCollectors which are only available to plugins. Because of this, I opted to put the /api/stats endpoint in the usage_collection plugin while keeping the /api/status endpoint inside Core. I had to duplicate a bit of the logic for translating KP status levels to the legacy status colors. LMK if you have a better idea or prefer to put these elsewhere.
• This introduces a new status format for /api/status which is disabled by default but can be opted-into with the v8format query parameter. My plan is to add a release note that the old format is deprecated and the default will change in 8.0/master. I will put up a quick follow up PR to change this in master after this merges.

Release Note

The /api/status endpoint response format is now deprecated and will change in 8.0.

This change only affects the status.* key of the JSON response, all other keys are unchanged:

Before

// curl mykibana.com/api/status
{
  "status": {
    "overall": {
      "state": "green",
      "title": "Green",
      "uiColor": "secondary",
      "icon": "success"
    },
    "statuses": [
      {
        "id": "elasticsearch@7.9.0",
        "state": "green",
        "message": "Ready",
        "uiColor": "secondary",
        "icon": "success"
      },
      {
        "id": "dashboard@7.9.0",
        "state": "green",
        "message": "Ready",
        "uiColor": "secondary",
        "icon": "success"
      },
      {
        "id": "visualize@7.9.0",
        "state": "green",
        "message": "Ready",
        "uiColor": "secondary",
        "icon": "success"
      },
      /** all other plugins */
    ]
  }
  /** all other fields are unchanged */
}

After

// curl mykibana.com/api/status
{
  "status": {
    "overall": {
      "level": "available",
      "summary": "All systems are ago!"
    },
    "core": {
      "elasticsearch": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
      "savedObjects": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
   },
   "plugins": {
     "dashboard": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
     "visualize": {
	    "level": "available",
	    "summary": "All systems are ago!"
	  },
      /** ...all other plugins */
    }
  },
  /** all other fields are unchanged */
}

Status codes and request parameters are unchanged.

In all 7.x versions, the old format will continue to be returned. In 8.0, the default format will change to the new format. Users of 7.10+ may opt-in to the new format by specifying the v8format=true parameter in the URL. This allows admins to update any external health checks at your convenience and provide a smooth upgrade experience to 8.0.

Checklist

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[afharo]

Just passing by because I've been working on elastic/beats#20956 and added some comments related to the /api/status API 🙂

[joshdover]

The test coverage on this endpoint is admittedly low. Since this will be owned by @elastic/telemetry and they have more knowledge of usage collectors (which is the bulk of the logic in this endpoint), I'm leaving this to them to flesh out more. This simply serves as a smoke test to ensure the request will complete successfully. There are also existing integration tests that test more permutations of auth & query parameter combos.

[mshustov]

Since this will be owned by @elastic/telemetry

Are you going to update CODEOWNERS in the following PR?

[joshdover]

They already own this entire plugin, so I don't believe any changes are needed to CODEOWNERS?

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 9f619ee

Build metrics

distributable file count

id	value	diff	baseline
default	45906	-12	45918
oss	27707	-11	27718
total		-23

History

• 💔 Build #75024 failed b327503
• 💔 Build #74983 failed 045f69a
• 💔 Build #74965 failed 88121e4
• 💚 Build #74833 succeeded a10e7e8
• 💔 Build #74641 failed 64576c2

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[afharo]

That was it! Thank you for fixing it!