[META] 7.16 Onboarding improvements around Empty States and Ingest

[snide]

Onboarding into Kibana from a brand new Cloud cluster can be confusing for new users. We have mutliple ingest flows between Beats, Agent, Clients and Sample Data. Further, our applications within Kibana don't provide a singular ingest URL for leading to these ingest flows. This meta ticket tracks all the times we'd like to address in the upcoming minor to fix some of these problems:

Source documents

Unified integrations

• Product Requirements: Unified integration view @alexfrancoeur
• Design document: Unified integrations view @dborodyansky
• Figma designs @dborodyansky

Add Agent

• Design document: Onboard Elastic Agent @dborodyansky
• Figma designs @dborodyansky

Integrations and Agent

• Implement Unified integrations
  • Provide final Figma designs @dborodyansky
    • Designs should include the ES endpoint available to the user (which was feedback from @mriley)
  • [Fleet] Unified Integrations view #93084 @alexfrancoeur @mfinkle @mostlyjason @thomasneirynck
  • Point empty states to integration manager
• [Fleet] Improve add integration to add agent flow #112626 (Fleet engineering team, @jen-huang point of contact)
  • Rationale and designs provided
• Address users running into empty assets
  • Fleet system package installs 15 Dashboards, 120+ Visualizations and 2 Kibana index-patterns #82851 @mostlyjason
• Add new integration types
  • Add Elasticsearch Clients as non-Elastic Agent integrations (MVP) #109281
  • Add App Search engines as non-Fleet integrations #109287
  • Add Workplace Search connectors as non-Fleet integrations #112842
• [Fleet] Integrations browse page doesn't retain/respect category in URL when first navigating to page #112871

Nice to have

• Add new integration types
  • Add Logstash as non-Elastic Agent Integration #112303

Empty States in Applications

• Empty state suggestions for ingest #107682 @snide @cchaos
  • [Security Solution] New Add Data Page #112142 @crowens @bradenlpreston
  • [Observability] New "No Data" screens #107709 @jasonrhodes
  • [Index Pattern Management] Show empty prompts in the page contents and don't auto-open flyout #109523 @ryankeairns @mfinkle
  • [META] [Kibana Analytics] Show consistent empty state when no data #111184 @ryankeairns

[mostlyjason]

@snide I think we can mark "Address users running into empty assets" as complete with the work Josh and others have done to improve the empty state detection logic. If the user views the dashboard or visualizations page without adding data, they will see a flyout to add data. We'll consider longer term improvements to avoid installing these assets here #108456

[snide]

@mostlyjason Partially. Can likely be improved by @ryankeairns' team as part of that empty page vs. flyouts issue. Clicking a link and then being redirected to another app is likely a bit confusing for the user, but the core logic check that @joshdover added should be enough for @ryankeairns' team to clean up that scenario (since it's the same problem happening all throughout the analytics app)

[elasticmachine]

Pinging @elastic/kibana-design (Team:Kibana-Design)

[alexfrancoeur]

@snide @dborodyansky I saw in the RFC doc a decision to add the elasticsearch endpoint to details pages. I believe we have the cloud ID for beats. Is this necessary for fleet integrations? If we were able to prioritize client libraries, it makes sense to add there. It should be simple enough to add, just curious what ingestion mechanisms you were thinking we need to focus on.

cc: @joshdover @thomasneirynck @mostlyjason

[snide]

@alexfrancoeur I think it's much simpler. Because we now skip this page in the setup, we need to print out the ES endpoint on the page (using ES directly is still extremely common). I'd suggest putting it possibly in the header or somewhere prominent of the integrations view:

I think we're looking at this as simply as:

Sorry for the rough screen, it was quicker than opening up Figma.

cc @mriley

[alexfrancoeur]

Ah wonderful, rough screens ftw. Makes sense to me! @dborodyansky please reach out to myself and @mostlyjason if you'd like any product feedback on the updated designs.

[dborodyansky]

@mostlyjason @snide @alexfrancoeur Thank you for the context around surfacing endpoint information. I added Dave's concept as well as some other potential ideas to the design decision doc to get further feedback.

[alexfrancoeur]

@snide assuming we get the unified integrations view in for 7.16, should be have a line item to update all empty states to point to it? We'll need to update the interstitial page and any other references to add data as well.

Thanks for updating the design doc @dborodyansky, it looks like there are some simple options we can explore. I'm going to play devils advocate for a moment. Given the priority of this work, I wonder if it's worth having a discussion with @philkra and the language clients team around introducing some simplistic in-product tutorials in parallel. In the latest comment it seems like we'd like to default to documentation (#109281 (comment)). The use case for an ES endpoint feels specific to clients and it'd be great to see the client libraries as discoverable as any other integration in 7.last.

[philkra]

@alexfrancoeur , in-product tutorials would be a preference, my understanding was that initially the only option was a link to documentation. If that a richer UX for language clients is the table for 7.16, the team would happily commit to providing the necessary artefacts.

[alexfrancoeur]

There is one other option that I suggested in #109281. We could leverage the add data tutorials in the short - mid term to provide a client-only experience that is not a package.

The bulk of these tutorials are in the home plugin, with some unique tutorials for EMS & APM.
https://github.com/elastic/kibana/tree/master/src/plugins/home/server/tutorials
https://github.com/elastic/kibana/blob/master/x-pack/plugins/maps/server/tutorials/ems/index.ts
https://github.com/elastic/kibana/blob/master/x-pack/plugins/apm/server/tutorial/index.ts

If the clients team was willing to contribute tutorials for each language client and / or a Kibana plugin to support them, it could be relatively simple to plug-in to this effort. Let me add a disclaimer here, that "relatively simple" statement is coming from a product manager. I know @thomasneirynck is doing some plugin mapping now to see how we can potentially fit APM / EMS tutorials into this unified view and it may not be as simple as I make it out to be. If it were, and @philkra and your team were able to support those tutorials in Kibana, we may be able to offer a more discoverable and user friendly experience leveraging variables such as dynamically adding an Elasticsearch endpoint, cloud ID, etc.

I suggest this not as a way to expand scope, but take advantage of an opportunity should it exist. If it's too much to consider for this first phase, let's start with doc links.

[snide]

@snide assuming we get the unified integrations view in for 7.16, should be have a line item to update all empty states to point to it? We'll need to update the interstitial page and any other references to add data as well.

Yes. This should be fairly easy. I've asked @cchaos to go ahead and merge and empty states (as they finish) for now (since they are still an additive improvement) and then we can simply change the links when we the integration view lands (either in the same view on new ones). Either way, I'll add the checkbox so we don't forget.

[alexfrancoeur]

@snide part of the original MVP requirements and design originally included an omni-present add data button as an entry point to this unified view (#94178). Is this still something we want? And if so, should we add as a "nice to have" for the unified integrations view work in 7.16.

[cchaos]

Adding the "Add Data" button in global navigation is quite trivial and we/I can add that pretty quickly. I do have some concerns around the proposed design, that I'll post in the related issue (#94178) but I see this as doable in 7.16 so long as we can agree on a single location for that button to refer too no matter which solution/app you are in.

[snide]

I'm closing this issue now that the main items are done for 7.16. Issues exist for the follow up items. Thanks team.