[Fleet] Unified Integrations view

[mostlyjason]

Describe the feature:
When integrations UI becomes the default experience for adding data in Kibana, we want to help users discover all the ways to add data to Elastic. We also want to have a unified landing page for adding integrations. This will provide a single place for new deployments to get started with the Elastic stack. It will promote adding their own data as the best way to get started.

Today, users have to check multiple places to identify which sources and use cases we support. They have to check integrations UI for agent integrations, add data UI for beats, enterprise search for content sources, apm for languages, and stack management for alert connectors. A unified page makes it easier for users to discover all the use cases we support.

To provide a unified list, we need to add support for non-Elastic Agent integrations like Enterprise search, Beats modules, Logstash, sample data, etc. This is also important when we have not yet published a GA integration for specific data source, or the user encounters one of the other limitations of Agent/Fleet. As a fallback, users can discover how to add data with Beats modules or Logstash plugins.

To provide discoverability, we could add link-type integrations. They provide tiles on the browse and search UIs for discoverability. They would include a title, description, icon, and a link to where users can add data for that source. It could link to an internal app in Kibana like Enterprise search or the Add data UI or a documentation page. They will not show up on the manage integration or data streams pages.

We can gradually transition those from link-type integrations to full integrations as we add support for new data sources. If we included them in the Elastic Package Registry, we could even release new GA integrations and replace the old ones out of band. This offers the teams creating integrations more flexibility to the control the timing of updates.

Describe a specific use case for the feature:

• As an enterprise search user, I would like a link to point me where I need to set up enterprise search after creating a new deployment.
• As a user of a service without a GA Integration available, I would like a link to point me to instructions to set up a Beats module or Logstash plugin.

Open questions:

1. At what granularity should we expose the integrations tiles when there are mutliple use cases per source? For example, Microsoft O365 supports both a logs module and workplace search. One solution is to expose two tiles: one for logs and one for search. Another solution is similar to what we have on https://www.elastic.co/integrations where there is a single tile with links for each use case.

Implementation

Phases

Implementing this has two primary phases, one implementing the base framework in the integrations app to support displaying, searching, and filtering cards registered by other Kibana plugins, followed by a second phase of items that can be tackled in parallel to add support for each type of card we need to add.

Base implementation:

• @thomasneirynck Add support for custom cards in the integrations app PR pending [Fleet] Add custom integrations API #112481
  • Add a new client-side registry to the Fleet plugin to allow plugins to register custom cards
  • Add logic for merging cards registered client-side with package list fetched from backend (likely around here)
  • Update cards UI to link to external apps for non-integration cards here.
• @thomasneirynck Add category selection and search term to integrations app router to support deep linking (PR pending [Fleet] Persist category and search in url on package page #111571)

Second phase, all top-level bullets should be able to be done in parallel:

Services

• @thomasneirynck Add support for beats module tutorials: [Fleet] Add categories to search response output package-registry#735
  • Expose a client-side API on home plugin for fetching tutorials (calling the internal server-side API for other plugins to use)
  • Add logic in Fleet client-side plugin that fetches the tutorials from home API and registers a card for each tutorial
  • Add option to Fleet's custom card API to allow a custom card to be displayed instead of an integration when the integration is not GA (more details below)
• @TinaHeiligers annotate all beats with category-assignment Annotates beats tutorials with custom integration categories #113565
• @joshdover Update permissions model to allow read-only access to Integrations app Allow users with read access to view Integrations app #113925
  • Allow users with "read" access to "Fleet & Integrations" app to access the Integrations app, but not the Fleet app [Fleet] Allow read-only access to integrations #94322
• @thomasneirynck Make categories not-fixed and expandable from both Kibana-end and EPR-end (pending final decision cc @alexfrancoeur) [Fleet] Have EPR register new categories / Show category counts #114429
• * Unified Integrations View - Count Clicks into Integrations @joshdover Unified Integrations View - Improve telemetry on integration funnel #113835 - nice to have

UI

• @thomasneirynck Update UX to clarify what an "installed" integration is ([Fleet] Rename tabs #111657) [Fleet] Add installed integration callouts #113893
  • Rename tab "Browse" -> "Browse integrations"
  • Rename "Manage" -> "Installed integrations"
  • Add tool tip to "Installed integrations" tab
  • Add explanation callout at top of installed list
• @clintandrewhall Add call out on integrations detail view for linking to corresponding beats module tutorials [Unified Integrations] Create Services, Storybook, Replacements Card; add to Fleet #113816
• @clintandrewhall Add Agent vs. Beats filtering UI [fleet] Add Integration Preference selector #114432
• @clintandrewhall Add deployment details callout (issue) [fleet][integrations] Provide Deployment Details on Cloud #114287
• @clintandrewhall *Update integration card display to use horizontal layout. [fleet] Adjust unified integration view to have better UI controls #114692
• Update add data links to navigate to the unified integrations view #114344
• Hide list pages of old add data tutorials [Getting Started] Remove legacy add data list view #94181
  • Hide tab bar on /app/home#/tutorial_directory
  • Add redirects from /app/home#/tutorial_directory to /app/integrations/browse
• Revise browse intro text and add link to docs. docs issue
• @thomasneirynck *Add support for custom cards in the global search results [here] (

  kibana/x-pack/plugins/fleet/public/plugin.ts

  Line 202 in 3cdced3

  deps.globalSearch.registerResultProvider(createPackageSearchProvider(core));

  ) replacement issue here Make all integrations searchable through the global search bar #115778
  • Bonus points if we can also solve at the same time: [Integrations] Missing icons in search bar #110978
  • We may also want to move this provider server-side if possible. Doing this would require that we implement the custom card registration APIs on the server-side instead of the client-side
• [Fleet, Workplace Search] Cannot link directly from unified integrations view to some Workplace Search integrations on less than Platinum license #115112
• [Fleet, Enterprise Search] Integration links lead to "Unable to connect" page if Enterprise Search backend is not running #115266

Additional Cards

• @thomasneirynck Add cards for linking to sample data [Fleet] Register Sample Data #113200
• @thomasneirynck Add APM card [Fleet] Add APM card to integration browser #113625
• @thomasneirynck [Fleet] Add cards for linking to file upload pages #113940 [Fleet] Add File upload #114934
• @yakhinvadim Register custom cards for App Search - MVP Add App Search engines as non-Fleet integrations - MVP #113825
• @yakhinvadim *Register custom cards for workplace search Add Workplace Search connectors as non-Fleet integrations #112842
• @thomasneirynck *Register custom cards for Language Clients [Fleet] Add language clients #113666 Add Elasticsearch Clients as non-Elastic Agent integrations (MVP) #109281
  • support pngs as icon-display in a card (required for language clients integration) [EDIT] not necessary. @dborodyansky regenerated to SVG
• [META] Update title and description based on guidelines for Beats integrations integrations#1883
• [META] Update title and description based on guidelines for Elastic Agent integrations integrations#1884
• Create title and description based on guidelines for App Search integrations #114332
• Create title and description based on guidelines Workplace search integrations #114333
• Create title and description based on guidelines Language Clients integrations #114335
• Create title and description based on guidelines for EMS, Sample Data and File Upload #114336
• *Register custom cards for App Search Add App Search engines as non-Fleet integrations #109287 (out of scope)

Bugs

• Fix breadcrumbs *Fix breadcrumbs on "add data" pages to emulate Integrations app #113434
• [Fleet] Handle scenario where EPR is not accessible #115564
  • Fallback to only showing custom cards
  • Add callout UI with information on how to configure on-prem EPR
• UI polish *[Fleet] Unified integrations - UI polish #113981
• [Fleet] [Integrations] Integration view search should query ALL integrations, rather than the category filter #114492
• going back to the ALL-category (in both tabs), does not apply the search-term (if it was present) (localSearch doesn't seem to be applied)

Tech Debt

• @thomasneirynck re-introduce category counts [Fleet] Have EPR register new categories / Show category counts #114429
• Do not throw errors when non-superusers use the global search NS Integrations results provider: avoid throwing when the user does not have superuser permissions #111094
• *API docs (prefer to wait on this after this stabilizes)
• *BUG: category seemed to be cleared on back-button press from other page [Fleet] Fix integration category redirect after loading categories #113812
• *rename custom_integrations-plugin to integrations :)
• *@clintandrewhall improve icon-typing
• search string not preserved in installed-integrations page [8.0 snapshot]: URL not updated as per query made in search bar under installed integration tab. #113518
• internationalization of categories
• [Fleet] Enterprise Search plugins are not included in the total count of custom integrations on CI #115108

* = stretch goals that are not necessary to swap over the default onboarding experience to the ingestions app

Service Design

Registry API

This API would be exposed on the Fleet plugin's client-side Setup interface (which is currently empty) so that plugins could call fleetSetup.integrations.registerCard

export interface IntegrationsServiceSetup {
  registerCard(card: CustomCard): void;
}

Card Interface

/**
 * Parameters that other plugins can use to define a custom card in the integrations UI.
 *
 * Designed to closely map to the {@link PackageListItem} type to make merging custom card results with registry search
 * results as seamless as possible.
 */
export interface CustomCard {
  /** User-visible title of card. Max characters XXX */
  title: string;

  /**
   * Machine-readable name of integrations.
   * For Beats modules, this should match the `moduleName` (eg. apache)
   */
  name: string;

  /** User-visible description of card. Max characters XXX */
  description: string;

  /**
   * Type of card to describe the behavior when a user clicks on card.
   * Defaults to `ui_link` which is the only currently supported type.
   */
  type?: 'ui_link';

  /**
   * UI path to link user to when clicking this integration.
   * Should not include basePath.
   * Only applicable to cards with `type: 'ui_link'`
   * Exmaple: /app/home#/tutorial/activemqMetrics
   */
  uiInternalPath: string;

  /**
   * List of available icons for this card.
   * If not present, UI will fallback to the EUI icon type for `logo${name}`
   */
  icons?: Array<{
    /** URL path to icon, should not include basePath */
    src: string;
    /** Mimetype of icon file. Only image/svg+xml is supported at this time */
    type: string;
  }>;

  /** Categories this card should be shown in. At least 1 category is required. */
  categories: PackageSpecCategory[];

  /** Used for mapping Beats modules tutorials to corresponding integrations */
  beatsModule?: {
    moduleName: string;
    type: 'logging' | 'metrics';
  };
}

Handling Beats

One tricky aspect is how to handle the mapping between tutorials for Beats modules and their corresponding integration. In summary, the behavior we want is:

• Custom cards for Beats modules should only be shown if the corresponding integration is not yet GA
• Integration details pages should show a callout to the corresponding Beats module tutorial(s), with a link to the filebeat (logging) and metricbeat (metrics) tutorials if available.
• Users without superuser access should see cards for all Beats modules, regardless of integration release status (@mostlyjason can you confirm this?)
• All of the logic above also needs to apply to the global search integration

In theory, we could provide a very generic API on the CustomCard interface to allow plugins to define some of this replacement behavior. Since we only need to support this for Beats at this time, I think it's safer if we provide an explicit field for this now (see above CustomCard.beatsModule) and handle this logic internally to the Integrations app. This makes the API less powerful but allows us to keep the scope low and puts less onus on external plugins to integrate correctly.

It'd be best if we could execute the bulk of this logic in a code path in the layer where we fetch the package list for integrations. The logic should likely follow something similar to this pseudo-code:

const getResolvedCards = async (isSuperUser: boolean, customCards: CustomCard[]) => {
  if (!isSuperUser) {
    return customCards;
  }

  const packages = await getPackages();
  const nonGaPackages = packages.filter(p => p.release !== 'ga');
  const gaPackages = packages.filter(p => p.release === 'ga');

  const filteredCustomCards = customCards.filter(c => {
    const matchingPackage = gaPackages.find(p => p.name === c.beats?.moduleName);
    return !matchingPackage;
  });

  const filteredNonGaPackages = nonGaPackages.filter(p => {
    const matchingCard = customCards.find(c => c.beats?.moduleName === p.name);
    return !matchingCard;
  })
  
  return [
    ...gaPackages,
    ...filteredCustomCards,
    ...filteredNonGaPackages
  ]
}

CC @mukeshelastic @sorantis

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[ruflin]

@mostlyjason I assume these integrations would also exist in the package-registry?

[mostlyjason]

@ruflin Yes that would make sense, I mentioned a benefit of including them there

[thomasneirynck]

#112330 is a POC for the new proposed flow. Not intended for review, just a workable demo to get a feel for the behavior and some of the edge-cases.

[dikshachauhan-qasource]

Hi @jen-huang

Could you please share the mocks for UI changes mentioned in ticket summary, so that we can also initiate work upon same.

As most of the changes described are mainly with UI so that will be helpful to cover relating testing.

Thanks
QAS

[joshdover]

@dikshachauhan-qasource We're still iterating on UX here, but we'll be sure to share it once things have been stabilized.

[joshdover]

I'm closing this issue now that the initial project is complete. Follow up issues are here:

Targeting 7.16

• [Fleet] Handle scenario where EPR is not accessible #115564
• [Fleet] Add UI integration tests for Integrations app #115788
• [META] Update title and description based on guidelines for Beats integrations integrations#1883
• [META] Update title and description based on guidelines for Elastic Agent integrations integrations#1884

Tasks for future iterations

• [Fleet] Unified integrations follow up tech debt #115784
• Make all integrations searchable through the global search bar #115778
• [Fleet, Workplace Search] Cannot link directly from unified integrations view to some Workplace Search integrations on less than Platinum license #115112
• [Fleet, Enterprise Search] Integration links lead to "Unable to connect" page if Enterprise Search backend is not running #115266