Fetch list of data products from datahub

[PriyaBasker23]

User Story

As a developer, I want to compile a comprehensive list of tables within the frontend for presentation to users.

Value / Purpose

The primary objective is to display categorised tables, organised by domains and data products.

Hypothesis

Implementing this functionality will facilitate the delivery of the frontend to users and support iterative development based on feedback.

Proposal

1. Implement the functionality to list data products by directly querying the datahub API for initial results.
2. Optionally, explore the use of a Python library for a more streamlined integration into the FastAPI layer at a later stage.
3. Note: The following steps are not mandatory but are considered for future enhancements.
   a. Develop a dedicated endpoint in FastAPI to handle the list of data products functionality.
   b. Invoke the Python function from the created endpoint to ensure seamless integration.
4. Maintain flexibility in the approach to accommodate iterative improvements and adjustments as necessary.

Definition of Done

The frontend should successfully display a categorised list of data products, organised by both domains and data products.

[murdo-moj]

datahub-project/datahub#4575

[murdo-moj]

GraphQL query. Both the interactive query builder on the datahub frontend and the docs on the data product entity are useful https://datahubproject.io/docs/generated/metamodel/entities/dataproduct/
https://datahub.apps-tools.development.data-platform.service.justice.gov.uk/api/graphiql

{
  search(input: {type: DATA_PRODUCT, query: "*", count: 500, start: 0}) {
    total
    searchResults {
      entity {
        ... on DataProduct {
          urn
          tags {
            tags {
              tag {
                urn
                properties {
                  name
                }
              }
            }
          }
          properties {
            name
            description
            numAssets
          }
          relationships(input: {direction: OUTGOING, types: "DataProductContains", count:500, start: 0}) {
            relationships {
              entity { ... on Dataset {properties { name } } }
            }
          }
          domain {
            domain {
              properties {
                name
              }
            }
          }
        }
      }
    }
  }
}

[murdo-moj]

This one is way faster. The subquery to parse the asset(dataset) names is what's taking the time

{
  search(input: {type: DATA_PRODUCT, query: "*", count: 500, start: 0}) {
    total
    searchResults {
      entity {
        ... on DataProduct {
          urn
          tags {
            tags {
              tag {
                urn
                properties {
                  name
                }
              }
            }
          }
          properties {
            name
            description
            numAssets
          }
          relationships(input: {direction: OUTGOING, types: "DataProductContains", count:500, start: 0}) {
            relationships {
              entity { urn }
            }
          }
          domain {
            domain {
              properties {
                name
              }
            }
          }
        }
      }
    }
  }
}

[murdo-moj]

I've got this working in python; just have to integrate the response into the frontend and put eg the api url in the right place in the config

[github-actions]

This issue is being marked as stale because it has been open for 60 days with no activity. Remove stale label or comment to keep the issue open.

[github-actions]

This issue is being closed because it has been open for a further 7 days with no activity. If this is still a valid issue, please reopen it, Thank you!