Add category-names to packages in search response / Add package-ids to category list in categories response

[thomasneirynck]

EPR service does not expose an API to determine which categories a package belongs to, or vice versa, which packages belong to a list of categories.

e.g. https://epr-snapshot.elastic.co/search is missing a categories-property for each package

[
  {
      "name":"activemq",
      "title":"ActiveMQ",
      "version":"0.0.1",
      "release":"beta",
      "description":"ActiveMQ Integration",
      "type":"integration",
      "download":"/epr/activemq/activemq-0.0.1.zip",
      "path":"/package/activemq/0.0.1",
      "icons":[
         {
            "src":"/img/activemq.svg",
            "path":"/package/activemq/0.0.1/img/activemq.svg",
            "title":"activemq",
            "size":"32x32",
            "type":"image/svg+xml"
         }
      ],
      "policy_templates":[
         {
            "name":"activemq",
            "title":"ActiveMQ logs and metrics",
            "description":"Collect logs and metrics from ActiveMQ instances"
         }
      ]
   },
   {
      "name":"auditd",
      "title":"Auditd",
      "version":"1.1.3",
      "release":"ga",
      "description":"This Elastic integration collects and parses logs from the Audit daemon (auditd)",
      "type":"integration",
      "download":"/epr/auditd/auditd-1.1.3.zip",
      "path":"/package/auditd/1.1.3",
      "icons":[
         {
            "src":"/img/linux.svg",
            "path":"/package/auditd/1.1.3/img/linux.svg",
            "title":"linux",
            "size":"299x354",
            "type":"image/svg+xml"
         }
      ],
      "policy_templates":[
         {
            "name":"auditd",
            "title":"Auditd logs",
            "description":"Collect logs from Auditd instances"
         }
      ]
   },
...
]

e.g. https://epr-snapshot.elastic.co/categories is missing package-ids for each category

[
   {
      "id":"aws",
      "title":"AWS",
      "count":1
   },
   {
      "id":"azure",
      "title":"Azure",
      "count":3
   },
   {
      "id":"cloud",
      "title":"Cloud",
      "count":2
   },
   {
      "id":"config_management",
      "title":"Config management",
      "count":1
   },
...
]

The absence of this link is a challenge when trying to build a facetted-browser that combines packages from multiple sources and has some deduping behavior going on based on package-overlap. Without knowing which epr-package belongs to which categroy (or vice versa), total category-counts cannot be calculated for all categories. For more context cf elastic/kibana#93084.

e.g. in this screen shot: facet-counts need to be based on packages of both EPR and other sources. These counts cannot be just additive, because sometimes EPR-packages will be replaced by multiple other packages (e.g. one EPR-package corresponding to 18 integrations, might be replaced by only 2 beats modules).

The work-around would be to send a search-request for each category (search?category=foobar). This introduces a significant performance hit and adds more code-cruft to clients. fwiw - already kibana needs to work-around this limitation by having to re-request all the packages each time a category is selected (Even though Kibana loads all packages by default first).

Ideally, this can be added to the policy-level.

https://epr-snapshot.elastic.co/categories?include_policy_templates=true will provides the relevant counts we want a user to see.

[mostlyjason]

@joshdover this came up as a requirement for the unified integrations view in 7.16. Is this the right approach and is it realistic to get in 7.16? It might be good to check with the ecosystem team because I know they have limited capacity. If not, a workaround could be duplicating and hard-coding this data in Kibana. The disadvantage is that we'd have to maintain/sync this information in two places instead of one until this is available.

[mtojek]

Hi @thomasneirynck ,

I think that the reasonable option would be Add category-names to packages in search response. The second one may cause issues at scale. If we have 100000... packages, we won't be able to simply list them on the categories view.

is missing a categories-property for each package

@ruflin Do you remember why this property is missing? Is it only because of minimizing the response body?

It might be good to check with the ecosystem team because I know they have limited capacity.

I guess that we need to build the official roadmap for the ecosystem team, so cc @akshay-saraswat @jlind23 . I think we should be safe to deliver this in 7.16.

[thomasneirynck]

thx @mtojek , adding it to search would work well imho.

[jlind23]

@mtojek sounds great indeed, we will review during our meeting for the 7.16. 👍🏼

[mtojek]

Let's first finalize #703 , then we can switch to this issue.