fix: use apply_chat_template to find turn boundaries and allow tool_calling field#2179

Merged

winglian merged 17 commits intomainfrom fix/chat_template_tokenizer_mask

Dec 17, 2024

+929-357

Commits on Dec 17, 2024

fix: use apply_chat_template to find turn boundaries and allow tool_calling field

NanoCode012
authored and
winglian
committed
fix: keys to include in turn

NanoCode012
authored and
winglian
committed
feat(doc): explicitly recommend setting train_on_eos and roles_to_train

NanoCode012
authored and
winglian
committed
fix: eos not being masked for tool due to template padding

NanoCode012
authored and
winglian
committed
chore: clear up docs

NanoCode012
authored and
winglian
committed
fix: default messages format, train_on_eos: turn, and train on all assistant msg

NanoCode012
authored and
winglian
committed
fix: properly warn if empty content

NanoCode012
authored and
winglian
committed
feat: parametrize chat_template tests to test different tokenizers

NanoCode012
authored and
winglian
committed
fix: set proper default for message key

NanoCode012
authored and
winglian
committed
fix: update defaults to match load function

NanoCode012
authored and
winglian
committed
fix: change defaults to use new

NanoCode012
authored and
winglian
committed
feat: add tool_calling dataset

NanoCode012
authored and
winglian
committed
feat: add tool_calling test

NanoCode012
authored and
winglian
committed
fix: add handling of edge case of mistral tokenizer with only system prompt

NanoCode012
authored and
winglian
committed
feat: refactor all test to follow source code

NanoCode012
authored and
winglian
committed
fix: remove unnecessary eos_token from phi35

NanoCode012
authored and
winglian
committed
fix test for phi3.5 since eos was dropped from chat_template
winglian
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: use apply_chat_template to find turn boundaries and allow tool_calling field#2179

Commits on Dec 17, 2024

fix: use apply_chat_template to find turn boundaries and allow tool_calling field

fix: keys to include in turn

feat(doc): explicitly recommend setting train_on_eos and roles_to_train

fix: eos not being masked for tool due to template padding

chore: clear up docs

fix: default messages format, train_on_eos: turn, and train on all assistant msg

fix: properly warn if empty content

feat: parametrize chat_template tests to test different tokenizers

fix: set proper default for message key

fix: update defaults to match load function

fix: change defaults to use new

feat: add tool_calling dataset

feat: add tool_calling test

fix: add handling of edge case of mistral tokenizer with only system prompt

feat: refactor all test to follow source code

fix: remove unnecessary eos_token from phi35

fix test for phi3.5 since eos was dropped from chat_template