This repository has been archived by the owner on Sep 24, 2024. It is now read-only.

Add the ability to see more granular metrics from wandb table artifacts #69

Closed

veekaybee opened this issue Feb 22, 2024 · 2 comments

Assignees

Contributor

veekaybee commented Feb 22, 2024 •

edited

Loading

We currently output evaluation results as table artifacts which gives us all the data but not as easy of a visual glance at evals.

We'd like to add these results to the table summary, which

Logs an artifact of the entire results json blob
Creates tables for individual tasks, and add those to the run summary

veekaybee self-assigned this

Contributor Author

veekaybee commented Feb 23, 2024

https://github.com/EleutherAI/lm-evaluation-harness/blob/40f2d19fc1c2b2c39313ad62697009272a758f3d/lm_eval/logging_utils.py#L205

https://github.com/EleutherAI/lm-evaluation-harness/blob/40f2d19fc1c2b2c39313ad62697009272a758f3d/lm_eval/logging_utils.py#L327

Member

aittalam commented Aug 20, 2024

Closing for now as we are mitigating wandb dependency

aittalam closed this as completed

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.