Improve search results for integrations

[alexfrancoeur]

Description

In 7.16, we defaulted to the integrations view for all ingestion options and added non-agent integrations for improved discoverability (#93084). While this is a significantly improved experience, search is increasingly becoming more valuable on the page as the number of integrations grow. As show by examples in elastic/uptime#351 and elastic/package-spec#209, there is much more we can do to provide search results on related and relevant terms not provided in the title or description of the integration.

User stories

• As an Elastic user, I'd like to search for my use case and find relevant integrations
• As an Elastic user, I'd like to search for related terms and find relevant integrations
• As an Elastic user, I'd like to search for categories combined with other terms to find relevant integrations
• As an Elastic package owner, I'd like to be able to define related terminology, use case and relevancy rankings as part of the package spec
• As an integration owner, I'd like to be able to define related terminology, use case and relevancy rankings for my integration

Capabilities

• Ability to provide free text search across integration titles, descriptions, categories and pre-defined terms associated with an integration
• Package owners have the ability to add terms related to their package and define prioritization of those terms
• Integrations owners have the ability to add terms related to their integration and define prioritization of those terms

Definition of done

As part of this enhancement, our end goal is to increase the click rate of integrations within this page. Users should be able to find what they're looking for. When we are confident that our search capabilities and results match expectations for our users and owners of integrations feel like all terminology associated with that integration is supported, we can call this feature done. If needed, we can provide a target goal for clicks into integrations based on search results.

[alexfrancoeur]

cc: @mostlyjason @LL6688

[mostlyjason]

As an Elastic package owner, I'd like to be able to define related terminology ... as part of the package spec

@alexfrancoeur @akshay-saraswat what about including the overview readme as part of the full text search? This already provides many related keywords.

As an Elastic package owner, I'd like to be able to define use case and relevancy rankings as part of the package spec

@alexfrancoeur @akshay-saraswat what about including some relevancy heuristics based on keyword location? Eg. Title > description > overview readme. This might have a lower maintenance overhead than manual ranking, and might prevent abuse by third parties in the future.

[alexfrancoeur]

Dropping in some wonderful feedback we've received from @serenachou to make sure we don't lose it. cc: @osmanis

First off, I love that we put this page together, I personally see a lot of ways for this to be useful to the solutions team but there's a few things that have me scratching my head on.

1. Search bar interaction with the category filters.

When using the search bar to find say all of the integrations that are relevant to Workplace Search or App Search, it's interesting that I can put in "Workplace Search" or "Zendesk" and it pulls up the tile, but the category facet on the side does not react to the search.

Example: Why do I still see all categories in the filters? Shouldn't it just be the category that the Zendesk integration is relevant to?

Then, it becomes even more odd, if I were to click into one of the listed categories while a search is active, It just leaves me stranded with this message and I have to remove the search term (another click) to then move to another integration.

The numbers associated with the categories also do not change in context of the search - if a tile fits multiple categories, I would expect that the number associated with the category would reflect how many tiles meet the search criteria and also are in those categories. They seem to be just static regardless of what you've input into the search.

There are also some small cues missing as to why the search results are returned, with missing hit highlighting.
Example: Why is Network Packet Capture returned for ze? Are we only matching on the name of the integration or other fields?

A small nit, I also would expect some auto complete, or perhaps auto suggestions in the search to highlight meaningful integrations to help guide the customer into more valuable data to be ingested?

In terms of the categories themselves, I am wondering whether there's any intention to add filters on other fields that I have seen in the other solutions tiles so I as a user can find integrations based on

• GA or Beta status
• Solution capability (e.g. all Enterprise Search or App Search related tiles)
• Licensing (Basic or Platinum)
• Deployment type (on-prem, cloud etc)
• features (search vs observability)

It feels like this page should be the superior version of https://www.elastic.co/integrations/data-integrations?solution=enterprise-search but the current iteration doesn't quite showcase the solution capabilities the way we need.

[LL6688]

@osmanis ; @dandreaj , fyi

[ghost]

Hi @alexfrancoeur,
While we were testing on the latest 8.1.0-BC6 build we have observed that while the user search Elastic Agent integration the search result doesn't show auto suggestion or find the integration based of its name.
However, the integration do show after the search but the user have to manually scroll the page to select the integration. Please find below our observation on the issue:

Screen-Recoding:

Browse.integrations.-.Integrations.-.Elastic.-.Google.Chrome.2022-03-04.18-16-38.mp4

Build Details:
VERSION: 8.1.0-BC6
BUILD: 50485
COMMIT: 4aaeda2
ARTIFACT: https://staging.elastic.co//8.1.0-6b89df21/summary-8.1.0.html

Thanks!

[serenachou]

@akshay-saraswat based on our conversation today, where should this type of request move to?