Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Misc-multilingual tasks #339

Merged
merged 91 commits into from
Oct 10, 2024
Merged

Misc-multilingual tasks #339

merged 91 commits into from
Oct 10, 2024

Conversation

hynky1999
Copy link
Collaborator

@hynky1999 hynky1999 commented Oct 2, 2024 •
edited
Loading

Goal

Fixes

  • correct hellaswag prompt, which I didn't implement correctly (didn't work if only ctx_a was submited)
  • fixes mlmm arc (previously was accessing incorrect field)
  • fixes formulation issue (previously we were not passing the formulation to template fc so it was using mcf all the time)
  • fixes metrics used (char/token used for all mcq + pmi for mmlu/arc and csqa)
  • fixes hf_avail splits (previously they were not defined and neither were few shot splits for some tasks which was causing errors)
  • Minor changes to adapters (so that tasks don't crash)
  • Switched arabic mmlu repo to official one MBZUAI/ArabicMMLU cc @alielfilali01
  • Fixes mkqa subsets (previously all variants used the binary subset)

New tasks

GK

  • openai_mmlu_tasks
  • turkish_mmlu_tasks
  • lumi arc
  • hindi/swahili/arabic (from alghafa) arc

Math

  • cmath
  • mgsm

Continuation

  • xcodah
  • xstory
  • xwinograd + tr winograd

RC

  • mlqa

Generative GK

  • mkqa
  • mintaka
  • mlqa_tasks
  • french triviaqa
  • chegeka

BoolQ tasks

  • acva
  • french_boolq
  • hindi_boolq

hynky1999 and others added 30 commits September 5, 2024 17:40
@hynky1999
add multilignaul dynamic generative metrics
5c69eb0
@hynky1999
Merge branch 'main' into geneartive_dynamic_metrics
39c4220
@hynky1999
Merge branch 'geneartive_dynamic_metrics' into config_templates
2a5cdca
@hynky1999
draft
2df9a08
@hynky1999
finish multichoice config
95729ee
@hynky1999
Merge branch 'main' into geneartive_dynamic_metrics
3aa0579
@hynky1999
update tokenizers + install nltk reqs
b8f90a9
@hynky1999
use punkt tab
f5a8717
@hynky1999 @NathanHB
Update src/lighteval/utils/imports.py
227f572 
Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>
@hynky1999 @NathanHB
Update src/lighteval/metrics/normalizations.py
d80b3ba 
Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>
fix imports
532bdad
remove unused import
75f7ac5
@NathanHB
Merge branch 'main' into geneartive_dynamic_metrics
f99e330
@clefourrier
Merge branch 'main' into geneartive_dynamic_metrics
92daf90
@NathanHB
Merge branch 'main' into geneartive_dynamic_metrics
f2a801d
finish implementation of templates + move stuff around
91d9d4f
resolve nits
9356cc6
when in rome do as romans do (handle error messages the same way)
0fbc731
@hynky1999
fix utils
fa1fa83
@hynky1999
Merge branch 'geneartive_dynamic_metrics' into config_templates
db36e16
@hynky1999
nicers tests + fix them
44aeecf
@hynky1999
nicer todo
2bff963
@hynky1999
add nice doscrings 📃
3c9eb21
@hynky1999
add even more docstring
4216ae2
@hynky1999
nit
d8f56b8
@hynky1999
fix test
f26e88c
@hynky1999
add multilingual to dev group
111d615
@hynky1999
merge nli, add languagees to literals
7ca4239
@hynky1999
translation literals
22eeddb
@hynky1999
add nli
7faaa8a
@hynky1999 hynky1999 changed the title Misc tasks Misc-multilingual tasks Oct 2, 2024
@hynky1999 hynky1999 changed the base branch from multilang_mqa_tasks to main October 2, 2024 15:09
@hynky1999 hynky1999 mentioned this pull request Oct 4, 2024
Closed
NathanHB
NathanHB approved these changes Oct 10, 2024
View reviewed changes
Copy link
Member

@NathanHB NathanHB left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small nits, otherwise looks good !

hynky1999 and others added 3 commits October 10, 2024 14:52
@hynky1999 @NathanHB
Update src/lighteval/tasks/default_prompts.py
1a7fd13 
Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>
@hynky1999 @NathanHB
Update src/lighteval/tasks/default_prompts.py
c299f89 
Co-authored-by: Nathan Habib <30601243+NathanHB@users.noreply.github.com>
@hynky1999
Merge branch 'main' into misc_tasks
de16271
@hynky1999 hynky1999 merged commit b018c9c into main Oct 10, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@NathanHB NathanHB NathanHB approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@hynky1999 @NathanHB @clefourrier