New metadata workflow endpoint that supports retrieving all workflow definitions, but only the latest version.

[alexmay48]

Please read our contributor guide before creating an issue.
Also consider discussing your idea on the discussion forum first.

Describe the Feature Request

In order to improve performance, we need a way to access workflow definitions that only loads in the latest version of the definitions. We have large amounts of definitions, and on top of the definitions, we have a lot of versions. The payload of the current GET "/metadata/workflow" endpoint returns more than a megabyte of data, and is slow. In an effort to lower payload size and increase speed performance, we could just grab the latest versions of the definitions.

In order to keep many of the filtering and sorting mechanisms in the definitions page, we would still need to keep load all definitions, but just loading the most recent version of the definitions would be very helpful in reducing payload size.

Describe Preferred Solution

Add a GET "/metadata/workflow/latest-versions" endpoint that will return List, but it will only be the definitions with the latest versions.

Describe Alternatives

None considered.

[manan164]

Hi @alexmay48 , there is also an endpoint to get the workflow definition by version. You can use that. GET /workflow/{name} .If we don't specify any version than the conductor will fetch the latest version.

[alexmay48]

Hi @manan164, the issue comes on the definitions page of the ui. It is loading all workflows, and ALL of their definitions in this call. This has affected the performance of our test system as we rapidly iterate on large worklows as we test them and have a large number of versions of a workflow. This means that we are pulling in megabytes of data to get all of the workflows. We have mitigated the issue by deleting and removing earlier workflow versions, but this would be very helpful. I am working on a branch right now that I will commit soon.

[manan164]

Hi @alexmay48 , there is also an endpoint to get the workflow definition by version. You can use that. GET /workflow/{name} .If we don't specify any version then the conductor will fetch the latest version.

Hi @alexmay48 , I am talking about a different endpoint here, Can you please try with the above endpoint? We need to know workflowName prior to invoking the endpoint. Currently, there is no exact endpoint that suffices your requirement. Feel free to raise PR.

[alexmay48]

@manan164 , yes that endpoint you mentioned works as intended. I went ahead and added a PR for this issue here: #3670