Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Alternative Worlds Prompts for Various Tasks and Benchmarks #925

Draft
wants to merge 59 commits into
base: main
Choose a base branch
from
Draft

Alternative Worlds Prompts for Various Tasks and Benchmarks #925

wants to merge 59 commits into from

Conversation

lintangsutawika
Copy link
Contributor

No description provided.

@lintangsutawika lintangsutawika mentioned this pull request Nov 13, 2023
Merged
@lintangsutawika lintangsutawika marked this pull request as ready for review November 14, 2023 13:36
@lintangsutawika lintangsutawika marked this pull request as draft November 14, 2023 13:36
@lintangsutawika lintangsutawika changed the base branch from big-refactor to main December 4, 2023 14:56
@lintangsutawika lintangsutawika changed the base branch from main to big-refactor December 4, 2023 14:57
@lintangsutawika
fixed file path
ec4cdfe
@lintangsutawika
remove style
09cac5d
@lintangsutawika
removed
6e220b1
@lintangsutawika
removed
edb21d2
@lintangsutawika
add alt prompts for truthfulqa
b9e5531
@lintangsutawika
added mmlu alt world prompts
0d5748b
@lintangsutawika
resolved merge conflict
6f4f9e1
@lintangsutawika
format
ec656bb
@lintangsutawika
reveresed os.walk
6fdecc6
@lintangsutawika
moved files
d5c6f0e
@lintangsutawika
removed comments
cd4ef5e
@lintangsutawika
allow doc_to_choice to be a single feature from a dataset
e96c648
@lintangsutawika
fixed list
3c5f347
@lintangsutawika
add styles 5 -8
5da401b
@lintangsutawika
fixed stderr for metrics like brier_score
4bff76d
@lintangsutawika
fixed style
5096772
@lintangsutawika
removed group and task from yaml template
4efa0b6
@lintangsutawika
fixed brier_score to allow multi-gpu inference
386d63e
@lintangsutawika
format
c027bc9
@lintangsutawika
Merge branch 'main' of https://github.com/EleutherAI/lm-evaluation-ha…
1b7d57c 
…rness into alt_worlds
@lintangsutawika
Merge branch 'main' of https://github.com/EleutherAI/lm-evaluation-ha…
56abc3a 
…rness into alt_worlds
@lintangsutawika
split to easy and challenge
f38c746
@lintangsutawika
add piqa
4277840
@lintangsutawika
add siqa
4654353
@lintangsutawika
add boolq
379bb7e
@lintangsutawika
fixed piqa ov
aa44be3
@lintangsutawika
solved merge conflict
0d1ef03
@RicardoDominguez
Copy link

Hi! Is there anything in particular holding back continuous metrics as in #969 (e.g., Brier score)? I'd be happy to contribute to this end.

@lintangsutawika
Copy link
Contributor Author

lintangsutawika commented Feb 23, 2024 •
edited
Loading

Thanks. I'll re-open the PR specifically to add Brier Score.

@lintangsutawika
Copy link
Contributor Author

@RicardoDominguez you can use brier score now. #969

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@haileyschoelkopf haileyschoelkopf Awaiting requested review from haileyschoelkopf

@StellaAthena StellaAthena Awaiting requested review from StellaAthena

@baberabb baberabb Awaiting requested review from baberabb baberabb will be requested when the pull request is marked ready for review baberabb is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@lintangsutawika @RicardoDominguez