[FEATURE] Wrap Up Profile API

[dhrubo-os]

Is your feature request related to a problem?

We started building Profile API in opensearch-py-ml repo: PR : #119

We need to wrap up this PR with adding unit test and further development (if needed) with addressing the comments.

[MaximilianMeiler]

I'll try my hand at this issue.

[dhrubo-os]

Thanks, assigned it to you.

[rawwar]

@dhrubo-os , I will be working on this issue. Thanks

[dhrubo-os]

Thanks, assigned to you. @rawwar

[rawwar]

@dhrubo-os ,
Having issues with getting task related data in the Profile API.

When I search for tasks using the following query, i get a lot of results.

 GET /_plugins/_ml/tasks/_search
{
  "query": {
    "match_all": {}
  },
  "size": 1000
}

One of the hits is as below

{
        "_index": ".plugins-ml-task",
        "_id": "Mhg2QYwBo5vXBZPtIh6Z",
        "_version": 3,
        "_seq_no": 24,
        "_primary_term": 1,
        "_score": 1,
        "_source": {
          "last_update_time": 1701901116071,
          "create_time": 1701901116057,
          "is_async": true,
          "function_name": "KMEANS",
          "input_type": "SEARCH_QUERY",
          "worker_node": [
            "TUeFQYfQRDWq5oc1-s1rhQ"
          ],
          "state": "COMPLETED",
          "model_id": "NBg2QYwBo5vXBZPtIh6g",
          "task_type": "TRAINING"
        }
      }

Now, when I try to get tasks using profile, i get nothing.

Query:

GET /_plugins/_ml/profile/tasks/

Result

{}

I tried giving "return_all_tasks": true and its still same empty result.

I also tried to use the task id from the above search result and its still an empty result

GET /_plugins/_ml/profile/tasks/Mhg2QYwBo5vXBZPtIh6Z

Result:

{}

Am i doing anything wrong here?

[dhrubo-os]

GET /_plugins/_ml/profile/tasks/

Returns runtime data for a specific task. You can string together multiple task_ids to return multiple task profiles

So I think, tasks are completed and not available in the memory?

[rawwar]

Returns runtime data for a specific task. You can string together multiple task_ids to return multiple task profiles

I interpreted this line as some runtime logs available for viewing at a later point. Reason being, for model profile, all of the model tasks are complete and they are returned as well.

[rawwar]

So I think, tasks are completed and not available in the memory?

I am not sure how we can test this. Since tests don't run in parallel, there's no way to test this if its in-progress data. I'll just ensure an OK response. Is that ok?

[dhrubo-os]

So I think, tasks are completed and not available in the memory?

I am not sure how we can test this. Since tests don't run in parallel, there's no way to test this if its in-progress data. I'll just ensure an OK response. Is that ok?

Yup, that should be fine for now.

[dhrubo-os]

Returns runtime data for a specific task. You can string together multiple task_ids to return multiple task profiles

I interpreted this line as some runtime logs available for viewing at a later point. Reason being, for model profile, all of the model tasks are complete and they are returned as well.

Those models tasks persisted in the index, so you can see. But for the profile api, we look into the memory. That's why outpout is different.

[rawwar]

@dhrubo-os , one information I could not get from documentation is the minimum OpenSearch version required for this API. Tests are successfully running for 2.7 and 2.11 for now. Hence, i haven't added any minimum version requirements for these tests

[dhrubo-os]

@dhrubo-os , one information I could not get from documentation is the minimum OpenSearch version required for this API. Tests are successfully running for 2.7 and 2.11 for now. Hence, i haven't added any minimum version requirements for these tests

We can keep min version as 2.7 for now. That should be fine.