Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance serving evaluation endpoints #595

Merged
merged 12 commits into from
Mar 1, 2024

Conversation

darrylong
Copy link
Member

@darrylong darrylong commented Feb 23, 2024

Description

This PR involves 3 changes to the serving functions:

  1. Add metric_user_results to evaluation results as user_result
  2. Modified /evaluate endpoint to accept evaluation data in the form of a json if data is included. Else, it will follow previous behavior of using feedback data added through the /feedback endpoint.
  3. query is removed from reponse

Sample request for /evaluate:

{
    "metrics": ["RMSE()", "NDCG(k=10)"],
    "data": [
        ["123", "1539", 1],
        ["123", "2", 1],
        ["124", "1", 1]
    ]
}

Response:

{
    "result": {
        "NDCG@10": 0.3175294778309396,
        "RMSE": 2.781925109617526
    },
    "user_result": {
        "NDCG@10": {
            "123": 0.20438239758848611,
            "124": 0.43067655807339306
        },
        "RMSE": {
            "123": 2.244862849697699,
            "124": 3.3189873695373535
        }
    }
}

Related Issues

Checklist:

  • I have added tests.
  • I have updated the documentation accordingly.
  • I have updated README.md (if you are adding a new model).
  • I have updated examples/README.md (if you are adding a new example).
  • I have updated datasets/README.md (if you are adding a new dataset).

@darrylong darrylong added the feature New feature/enhancement request label Feb 23, 2024
@darrylong darrylong requested a review from tqtg February 23, 2024 09:41
@darrylong darrylong self-assigned this Feb 23, 2024
@tqtg
Copy link
Member

tqtg commented Feb 23, 2024

I think we can remove "query" from the response to minimize the bandwidth because "data" could be quite significant.

@tqtg
Copy link
Member

tqtg commented Feb 23, 2024

Also, the returned user_result contains mapped user indices. Shall we try to map them back to the original user IDs, using the mapping in train_set, for it to be consistent with data in the request?

@darrylong
Copy link
Member Author

Also, the returned user_result contains mapped user indices. Shall we try to map them back to the original user IDs, using the mapping in train_set, for it to be consistent with data in the request?

Now using mapped user indices, new response is updated in main post.

@darrylong darrylong changed the title Enhance Serving Evaluation endpoints Enhance serving evaluation endpoints Feb 27, 2024
@darrylong darrylong marked this pull request as ready for review February 29, 2024 16:07
@darrylong darrylong merged commit 37db3c9 into PreferredAI:master Mar 1, 2024
16 checks passed
@darrylong darrylong deleted the enhance-serving-eval branch March 1, 2024 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature/enhancement request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants