-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increment max penalty + complement task with layout criteria #86
Conversation
else: | ||
layout_criteria = MatchLayoutCriteria( | ||
layout_type=LayoutMatchTypeEnum.NUMBERED_LIST, | ||
penalty=0.5, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no text here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it will get the default text Your response should be ordered in format of numbered list
. The bullet point criteria would list like "unordered list", that's why I override it to be "bullet points"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks fine, but do we need extra description on the numbered list?
else: | ||
layout_criteria = MatchLayoutCriteria( | ||
layout_type=LayoutMatchTypeEnum.NUMBERED_LIST, | ||
penalty=0.5, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
penalty=0.5, | |
penalty=0.5, | |
text="Your response should be ordered in format of numbered points.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It will get it by default Your response should be ordered in format of numbered list
Extra comment: what are the existing layout criteria used by qg and qa?
|
qg:
qa:
|
Do we specify the number of list items? |
and why not update the qa criteria to sometimes allow lists? this is actually reasonable behaviour at this point |
no, it's a binary verification, it was too last minute to bring this sort of complexity to the release imho |
because most of the miners already return in list layout already since they are leveraging the qg+qa / key takeaways exploits, so forcing it to be simple punishes those type of exploits |
good stuff. thanks! |
Note: The tasks of qg and qa already follow a defined layout, so the expansion was performed on the summarization task