Questeval

[padipadou]

• Adding a new metric, QuestEval, which is using source and / or references to evaluate models.

• This metric needs to know on which task the model is evaluated, if the task is not specified in data, QuestEval will use a general model to evaluate which might not fit well (e.g. Textual QA models dont work well on tables)

• minor typos in README

[tuetschek]

@padipadou : Thanks, looks mostly good to me! I'd have two points:

• It would be good to add tests, now that we have them for the other metrics – would that be possible?
• I'm not completely sure about adding the "task" attribute into the predictions set. Maybe we want the different versions of QuestEval exposed as different metrics (would there be a situation where you'd want to measure more of them?). What does @sebastianGehrmann think?

One small note: I made some changes to run_metrics.py recently and moved most of its code to gem_metrics/__init__.py so that all of GEM metrics can be installed by pip (and you get a script called gem_metrics in your $PATH). This is the probable cause of the merge conflict – I guess I can look into that for the merge.

[padipadou]

Hello @tuetschek ,

About unit tests: great indeed, I will add them this week (after or before the merge request accepted, up to you)
About adding the "task": BERTScore for instance requires the language and according to it use a multilingual model or not, still it is the same task. The same way, QuestEval uses different QA models if the input is a table or a text, but it is still the same metric. So I think at some point, this information will be required anyway.

Tell me what you think ! :)

[padipadou]

For now:

• I rebased my branch onto current main
• I modified setup.py file allowing to be enough generic to tolerate package name to be different from repo name.
• tests are not added yet.

[sebastianGehrmann]

Thanks for all this work! I wanted to chime in wrt task in prediction set.
We already have that information from the name of the predictions, since every task prefix is the name of the dataset. In fact, our website already has this file: https://github.com/GEM-benchmark/GEM-benchmark.github.io/blob/main/web/results/eval_config.json which links summarization to xsum etc.

Could we somehow use this file to determine the task? (you will have to add QuestEval here as well btw to get it rendered on the website)

In the same vain, @tuetschek, we could get rid of the language tag and move that to the eval_config.json.

I also just noticed that our test outputs don't have a gem_id along the predictions. Since TFDS shuffles files by default, we need to retain those to get the predictions in order and to allow subset evals (see #38)

[padipadou]

• Unit tests are added.
• I will look also at the website's repo to see what should I do.
• From the beginning, I used the way explained in README to test my code, e. g by passing a particular file (and therefore, unknown from the eval_config point of view) to the metric. Then, in this particular case, it would be impossible to guess the language / task of the given predictions.
• Ok for the gem_id, but I guess QuestEval metric is not particularly concerned by this ?

Thanks for the quick and clear answers !!

[sebastianGehrmann]

Hi @padipadou, thank you for all the updates! My comment wrt gem_id was more targeted at @tuetschek since this is a functionality we need independent from metrics.

I think it definitely makes sense to refactor the current setup of the framework to just look up a task in the eval_config where it can have a specified language + high level task.

[tuetschek]

@padipadou : Thanks, looking good!
@sebastianGehrmann : So we'll have both task and language, but they'd default to whatever is said in the config file? Do you have the config file somewhere? Because that's not used by the metrics code at the moment.

Re. gem_id, that's completely unrelated to this, right?

[sebastianGehrmann]

yes, the gem-id argument is unrelated, I just noticed that our test examples do not include them. Hence my initial comment.

Wrt config file - you can find a version at https://github.com/GEM-benchmark/GEM-benchmark.github.io/blob/main/web/results/eval_config.json
It needs to be updated with the newer metrics (Emiels, NUBIA, and QuestEval), but in the best case we'd use the same file for both website and eval framework.

[tuetschek]

Thanks! I'll open a new issue regarding the config file.