Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allowing a single prompt to use several formats for one eval. #398

Merged
merged 3 commits into from
Nov 22, 2024
Merged

Allowing a single prompt to use several formats for one eval. #398

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
247ddd4
adding index
clefourrier Nov 19, 2024
9036896
Merge branch 'main' into use_several_formats
clefourrier Nov 19, 2024
e8ac4a1
Merge branch 'main' into use_several_formats
clefourrier Nov 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/lighteval/tasks/lighteval_task.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -289,11 +289,13 @@ def _get_docs_from_split(self, splits: list[str], few_shots=False) -> list[Doc]:

docs = []
for split in splits:
for item in self.dataset[split]:
for ix, item in enumerate(self.dataset[split]):
# Some tasks formatting is applied differently when the document is used for fewshot examples
# vs when it's used for the actual prompt. That's why we store whether we are currently using the
# doc for a fewshot sample (few_shots=True) or not, which then leads to the creation of a different Doc.
item["__few_shots"] = few_shots
# Some tasks require to know which is the current item index in order to apply a different prompt template
item["__index"] = ix
cur_docs = self.formatter(item, self.name)
if cur_docs is None:
continue
Expand Down
Loading