Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hard-negative flag to include similar challenging negatives on triplets #856

Merged
merged 2 commits into from
Aug 6, 2024

Conversation

plaguss
Copy link
Contributor

@plaguss plaguss commented Aug 6, 2024

Description

Adds a flag to GenerateSentencePair to generate hard-negatives that are more challenging to distinguish from the positives. It applies only when triplet=True. The distinction between the prompts is the following:

while the negative sentence
- can use similar words but must not be related to the anchor sentence.
+ is a 'hard negative' that meets the following criteria:\n
+ - Uses similar keywords or phrases as the anchor sentence\n
+ - Has a similar grammatical structure or syntax\n
+ - Is not related to the anchor sentence, but could be mistaken for it\n
+ Try to create a negative sentence that would be challenging for a model to distinguish 
+ from the positive sentence
You must output only two new sections: `## Positive` and `## Negative`."

By default, the standard hard_negative=False will be used.

from distilabel.steps.tasks import GenerateSentencePair
from distilabel.llms import InferenceEndpointsLLM

generate_sentence_pair = GenerateSentencePair(
    triplet=True, # `False` to generate only positive
    action="query",
    context="Argilla is an open-source data curation platform for LLMs.",
    hard_negative=True,
    llm=InferenceEndpointsLLM(model_id="meta-llama/Meta-Llama-3.1-70B-Instruct"),
)

Closes #789

@plaguss plaguss added this to the 1.3.0 milestone Aug 6, 2024
@plaguss plaguss requested a review from gabrielmbmb August 6, 2024 07:43
@plaguss plaguss self-assigned this Aug 6, 2024
@plaguss plaguss linked an issue Aug 6, 2024 that may be closed by this pull request
Copy link

github-actions bot commented Aug 6, 2024

Documentation for this PR has been built. You can view it at: https://distilabel.argilla.io/pr-856/

Copy link

codspeed-hq bot commented Aug 6, 2024

CodSpeed Performance Report

Merging #856 will not alter performance

Comparing sentence-pair-hard-negatives (e5e3f27) with develop (4e1f2bc)

Summary

✅ 1 untouched benchmarks

Co-authored-by: Gabriel Martín Blázquez <gmartinbdev@gmail.com>
@plaguss plaguss merged commit ff3f484 into develop Aug 6, 2024
7 checks passed
@plaguss plaguss deleted the sentence-pair-hard-negatives branch August 6, 2024 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Improve GenerateSentencePair to include hard negatives
2 participants