Implement a UI within Stack Monitoring to help users migrate to MB shipping/collecting

[ycombinator]

We want to introduce a UI that helps the user start or migrate to using Metricbeat as shipper and collectors of monitoring data. The effort involved here is multi-step and are as follows:

Note: This should disabled on cloud as there is no way to install a beat there yet.

Step 1

PR UP: #34871

Provide infrastructure for product-specific status detection. We will need logic that understands how to detect a product in various phases, including:

• Not installed
• Not installed (we don't see monitoring) but we think it might exist
• Installed, but monitoring not enabled/configured
• Installed with monitoring, but using internal collection
• Installed with monitoring, using internal and MB collection
• Installed with monitoring, using MB collection

Part of this effort involves mid-flight detection, where they are in the process of upgrading but maybe missed a step or forgot to disable internal collection.

Step 2

Part 1

PR UP: #35228

Introduce product-specific migration steps, most likely utilizing a flyout for instructions. Each product will need a different set of steps and we should organize these into separate files, similar to how the "Add Data" feature is done. Here is what we're thinking:

Part 2

PR UP: #39832

Introduce setup steps for a net new user experience where we do not detect any monitoring data and assist the user in setting up Metricbeat collection with each stack product we think they are using. We know this information from part 1 and we plan to deliver a UI exactly the same as the cluster listing page we have now. More to come here

Step 3

PR Up: #40442

Update the core parts of this work with progress made in enabling Metricbeat collection in other stack products, specifically logstash, beats and apm.

We also want to add user-friendly UX flows too, specifically:

• Add UX to help user access instructions (something like Setup monitoring button at the bottom of listing page for each stack product)

And lastly, we need to disable this feature for cloud as installing and running a beat on cloud is not currently supported.

Step 4

This step includes other improvements that come up through testing. I will be sure to include them so this can serve as a master copy of what this feature is.

• Add better UX for ES migration if the ES cluster is the one Kibana is talking to
• Share code better, but ensure it's consistent if possible with our areas of the codebase
• Follow up on EUI bug
• We should fetch setup data at the same interval as the other data for the page. This will allow the setup mode UIs to automatically update periodically which is more seamless experience for the user. As a result of changing this, we may not need a Check Data button in the flyout anymore. This comment started the discussion.
• Ensure this PR is applied to logstash, beats and APM panels

Step 5

Introduce a "Setup Mode" for the monitoring UI code base that, when toggled, will add/change the UI in a way that's more helpful for setup rather than monitoring itself. This mode will show the user migration status on their monitoring products, or it will also serve as a way to start monitoring new products as well. The resulting UI here will leverage the existing UI as much as possible. Here is a sample screenshot:

In addition to this UX/UI work, we also need to add the following in this step:

• Tests
• Final approval of copy from @gchaps
• Disable /no-data flow and default to /loading flow

Conclusion

These steps will require at least one piece of input from the user, which is the network address of the monitoring url. We can do our best to auto-detect what we think this should be, but we need to allow the user to override and use a different address. This input will then be used for the instructions, in place of an empty configuration string for monitoring hosts/url.

It's also possible we'll need to find a solution for credentials. I'm not sure what this will look like, but we might need to solve for this somehow.

[ycombinator]

Original comment by @chrisronline:

We should iron out how we're going to handle this:

In this scenario we don't have any existing information about the user's stack setup. However, by making live API calls we can at least figure out information about their Kibana instance and Elasticsearch nodes. For Logstash and Beats instances, we can try to make some educated guesses by looking for the existence of certain indices (logstash-, filebeat-, metricbeat-, auditbeat-, APM indices, .logstash, .management-beats).

Let's list out the specific ways we can "best guess" that a product does exist.

Beats

• *beat-* index existing
• .management-beats* index existing

Logstash

• logstash-* indices existing
• .logstash* indices existing

APM

• apm-* indices existing

[elasticmachine]

Pinging @elastic/stack-monitoring

[chrisronline]

This has been finished and all merged into master