Prompt Composition

[thomasfifer]

The AI (vicuna-13b-4bit) has trouble following and staying on the exact JSON format needed. I found some changes that appear to allow it to work correctly for a while longer.

I've changed constraint 4 to

Exclusively use the commands listed in double quotes e.g. "command name" with their associated args

since the AI will otherwise sometime substitute args from other commands or omit certain required args.

I've changed the performance evaluation paragraph to

Only respond in JSON format as described below and nothing else! This rule the most important as you will not be able to complete any other objective without responding in the correct JSON format! You also need to only respond with one JSON reply as described below at a time since the assistant can only complete one task at a time.

to help keep the AI replying with JSON since it sometimes reverts back to conversational text. It has also tried sending multiple commands at once breaking things.

[thomasfifer]

I'm currently seeing the command change to its description after some time, for example "Google Search" instead of "google". I think this could be a code change though to help out the AI when it gets things wrong to find a best match command corresponding to what the AI is trying to accomplish.

[rhohndorf]

I prompted the online version of open assistant 30b (just copy paste. I believe there is no public api yet) with example prompts from auto-llama. That worked very well. The 30b 4bit ggml version downloaded from user black-engineer on huggingface produced no usable output for whatever reason.
But i believe we should concentrate on the oasst models.

[wiseman-timelord]

Telling it to "Make sure to be using Powershell commands in the command prompt" helped in file operations, I know this may just be for windows, but, sure there is a relating contextual prompt for given OS.

Found the file with constraints, cheers.

[newbie-lad]

Assistent Reply If you complete a task, add "task\_complete" in the top level of your JSON and include the reason for completion in the "reason" field.

{
    "thoughts":
    {
        "text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    },
    "task\_complete": {
        "reason": "Reason for completion"
    }
}
### Assistant: {
"thoughts": {
"text": "I need to start by researching ...",
"reasoning": "I want to make sure I have a good understanding ..."
},
"command": {
"name": "Google search",
"args": ["..."]
}
}

json If you complete a task, add "task_complete" in the top level of your JSON and include the reason for completion in the "reason" field.

{
    "thoughts":
    {
        "text": "thought",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "speak": "thoughts summary to say to user"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg name": "value"
        }
    },
    "task_complete": {
        "reason": "Reason for completion"
    }
}
### Assistant: {
"thoughts": {
"text": "I need to start by researching ...",
"reasoning": "I want to make sure I have a good understanding ..."
},
"command": {
"name": "Google search",
"args": ["..."]
}
}

json loads error Expecting value: line 1 column 1 (char 0)
Warning: Failed to parse AI output, attempting to fix.
 If you see this warning frequently, it's likely that your prompt is confusing the AI. Try changing it up slightly.

Help me understand what is triggering [loop]
------------ JSON FIX ATTEMPT ---------------
:-)

[Vitorhsantos]

I'm also trying to find the source of these unwanted messages that precede the JSON code. I'm starting to think it has something to do with the model used, in my case I'm using ggml-vicuna-13b-4bit-rev1.bin. What model are you using?

[newbie-lad]

Well, obviously, we should be using Vicuna:

Overall the Vicuna model performed much better than the original LLama model in terms of answering in the required JSON format and how much sense the answers make. I just couldn't get it to stop starting every answer with ### ASSISTANT. I am very curious to hear <...>

And obviously, it's partly model's “fault”. (You can only expect this much from 13 B model.)

---

If we could find the cruel-pit prompt that is causing this, we could at least start “debugging” it.

[Vitorhsantos]

I've tried several different prompts to try to remove these messages (and the ### Assistant) and all to no avail. I had some success with processing the outputs to get just the JSON code. I had to change some commands to make sense with the commands the vicuna understood from the prompt. Anyway, I'm having some progress, I'll post more about it soon.

[newbie-lad]

git remote add thomasfifer https://github.com/thomasfifer/Auto-Llama-cpp.git
git fetch thomasfifer
git checkout thomasfifer/vicuna-13b

Originally posted by @thomasfifer in #10 (comment) is a good patch (it solves ### Assistant:).
Please, note the line:
"args": "input": "<search>" (or any other JSON that is poorly generated)
By no means will it not trigger errors 😄 (After colons should object or array follow, not strings.)

That has to be “debugged” — confusing (to a model) prompt that lead to the wrong output.
Although, I have no idea how to get such prompt. What is causing the model's confusion?

[DiegoSVilla]

I havent used this yet, but I have been trying to make llama follow strict json functions and... well, its as frustrating as it sounds out loud, lots of face spit.

My prefered models where wizard variants, maybe you should try,
#models/wizardlm-30b.ggmlv3.q4_K_S.bin
#upstage-llama-2-70b-instruct-v2.ggmlv3.q2_K.bin
this are kinda big, but the smaller variants did well.

Second, I saw that the #Assistant bull is the first the model abandons if you end the rprompt with "{"

If you do that its going to respond a json even if you dont want it to. [ also works. then when the prompt finishes you balance the output with the token

[rhohndorf]

llama.cpp has got grammar enforced json generation now ggerganov/llama.cpp@3ebb009
This is still very experimental and I haven't tried it myself, but this forces the llm to generate output with a certain structure.