Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide agent with details on why tool call was rejected #11

Open
bienehito opened this issue Nov 25, 2024 · 0 comments
Open

Provide agent with details on why tool call was rejected #11

bienehito opened this issue Nov 25, 2024 · 0 comments
Labels
bug Something isn't working enhancement New feature or request

Comments

@bienehito
Copy link

bienehito commented Nov 25, 2024

The get_result_message_simple currently validates function call arguments before calling the tool_fn and returns a generic reject_arguments_prompt (The previous tool call included unexpected arguments or argument types. Please try again with the correct arguments, or attempt a different action) if the validation fails. This causes degradation in performance of llm agents on tasks since it doesn't have information on how to correct the previous submission.

For example, given task instructions "Give your answer as a json object with a single key, "answer", whose value an integer.", llm produces a function call:

submit:
{
  "answer": 80
}

instead of

submit:
{
  "submission": "{\"answer\": 20}"
}

If the agent were given a more detailed message, such as "unknown submit function argument answer" or "missing required submit function argument submission" then the agent could self-correct. Currently most of the time the agent gets stuck in the loop eventually running out of token budged.

Important question is whether this would give an unfair advantage to the agent over a human performing the same task. I would argue that it will not when considered in the following scenarios:

  1. Human is submitting results in the same way as llm agent, e.g. by writing submit: { "submission": "{\"answer.... In this scenario, IMO human would struggle as much as the llm agent given the non-informative feedback.
  2. Human is submitting results via cli submit "{\"answer... since if the human mistyped the submit command they would be given an informative feedback allowing them to correct the submission.

P.S. Looking at the get_result_message_simple code, I don't see where presence of required arguments is validated.

@chriscanal chriscanal added bug Something isn't working enhancement New feature or request labels Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants