Bug in instruction_following_eval

[wlhgtc]

@lehougoogle
Thank you for your excellent work on the instruction-following ability. However, I've noticed a bug in your code.

In the build_description method within your code, specifically when handling the Instruction type RepeatPromptThenAnswer, a parameter named prompt_to_repeat is required. However, I observed that in your kwargs, this parameter is missing.

Although there is an attempt to address this issue through this line, it seems there's a mismatch in the parameter name (prompt_to_repeat should be used instead of prompt).

I managed to resolve this issue in my implementation with the following code:

if isinstance(instruction, RepeatPromptThenAnswer):
    instruction.build_description(prompt_to_repeat=record["question"])
else:
    instruction.build_description(**record["kwargs"][index])

[wlhgtc]

And by the way: These two methods((removeprefix and removesuffix)) were introduced in Python 3.9.
Code will not work in older version.
Maybe you should declare it in the readme?

[geronimi73]

just noticed the same. it seems that the prompt_to_repeat is in the first line of prompt.
I fixed it with

if isinstance(instruction, instructions.RepeatPromptThenAnswer):
  instruction.build_description(prompt_to_repeat=inp.prompt.split("\n")[0],**inp.kwargs[index])
else:
  instruction.build_description(**inp.kwargs[index])

.. in test_instruction_following_loose() and test_instruction_following_strict()

[lehougoogle]

Sorry. There's a stupid bug. The prompt set is wrong. Fix coming in within 12 hours.

[lehougoogle]

Hi all,
The prompt sets are updated. It should work now!