llm prompt --schema X option and model.prompt(..., schema=) parameter

[simonw]

Refs:

• Ability to specify a schema for structured output for models that support it #776

TODO:

• Logging the schema that was used in a database (ideally without duplicating it too many times)
• Schema should be shown in llm logs --json
• Schema should be shown in llm logs Markdown
• Docs on how to use this
  • Note in the docs about schema compatibility - how different models may have slightly different JSON schema features that they support (kind of sucks but I'm stuck with that I think) - in the future maybe describe a subset of JSON schema that LLM supports and ensure that all plugins support that subset
• Get the thing where you can pass a Pydantic BaseModel to prompt(..., schema=) to work (or cancel that feature)
• Automated tests
• Docs on how to write plugins that support schemas

[simonw]

I'm going to save schemas in the DB and refer to them from a new schema_id column on the responses table.

That schema_id will be a hash of the JSON schema itself, after a json.dumps(schema, separators=(",", ":")) - note that I'm preserving the key order because that matters for models like Gemini.

I'm going to try blake2b for this, available in Python since Python 3.6. I can use it to get the same length of ID as MD5 but it's more cryptographically secure (not that it matters here, but I do like avoiding hash functions which trigger security audits like that FIPS thing).

>>> hashlib.blake2b(content.encode(), digest_size=16).hexdigest()
'6bc5ba84cec9e6ceff3b65c8828b82b2'

[simonw]

For the Markdown output I'm going to spot if a prompt had a schema AND the response appears to be valid JSON and pretty-print and fenced-code-block that response.

[simonw]

Here's example output from llm logs -n 1 now:

2025-02-26T22:07:08 conversation: 01jn24sajg7wz3aqwnnkcdx310

Model: gemini-2.0-flash

Prompt:

invent three dogs

Schema:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "dogs": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string",
            "minLength": 1
          },
          "bio": {
            "type": "string",
            "minLength": 1
          }
        },
        "required": [
          "name",
          "bio"
        ],
        "additionalProperties": false
      }
    }
  },
  "required": [
    "dogs"
  ],
  "additionalProperties": false
}

Response:

{
  "dogs": [
    {
      "bio": "A small, fluffy dog with a penchant for finding lost objects. Known for its unwavering loyalty and ability to cheer up anyone with its playful antics.",
      "name": "Lucky"
    },
    {
      "bio": "A large, muscular dog with a gentle soul. Renowned for its bravery and protective nature, it's always ready to defend its loved ones.",
      "name": "Thor"
    },
    {
      "bio": "A sleek, intelligent dog with an insatiable curiosity. Excels at problem-solving and agility, always eager for a new challenge.",
      "name": "Spark"
    }
  ]
}

[simonw]

When replying to an existing thread with llm -c you have to remember to repeat the schema or this happens:

llm -c 'invent three more'

Okay, here are three more dog characters:

{
  "dogs": [
    {
      "name": "Patches",
      "breed": "Mutt (likely a mix of Dalmatian, Jack Russell, and something else)",
      "appearance": "Small-to-medium sized with short, wiry white fur covered in mismatched black and brown spots. One floppy ear, one that stands straight up. Big, expressive brown eyes.",
      "personality": "Energetic and mischievous. Loves to dig, chase squirrels, and 'help' with gardening (which usually means making a mess). Extremely friendly but a little clumsy.",
      "backstory": "Found as a stray near a farm. Always seems to be looking for adventure."
    },
    {
      "name": "Professor Woofington (Prefers 'Professor')",
      "breed": "Bernese Mountain Dog",
      "appearance": "A massive, fluffy tri-color dog. Carries himself with an air of dignity, despite constantly shedding. Wears a tiny pair of (non-prescription) reading glasses perched on his nose.",
      "personality": "Calm, thoughtful, and surprisingly well-read (enjoys listening to audiobooks). Believes in naps as a form of intellectual stimulation. Secretly loves belly rubs but pretends to tolerate them for the sake of his humans.",
      "backstory": "His human is a retired university professor who treats him as an equal. The dog genuinely seems to have absorbed some of the professor's knowledge."
    },
    {
      "name": "Shadow",
      "breed": "Belgian Malinois",
      "appearance": "Sleek and athletic with short, fawn-colored fur and a black mask. Intensely focused eyes.",
      "personality": "Quiet, observant, and intensely loyal. Highly intelligent and driven. Doesn't waste energy on unnecessary barking or playing. Always alert.",
      "backstory": "A former police dog who was injured in the line of duty and now lives with his former partner. Still yearns to be working."
    }
  ]
}

As opposed to:

llm -c 'invent three more' --schema "$(cat dogs.schema.json)"

{
  "dogs": [
    {
      "bio": "A tiny chihuahua with an enormous ego. Thinks he's a Rottweiler and isn't afraid to take on dogs ten times his size. Has a surprisingly loud bark.",
      "name": "Napoleon"
    },
    {
      "bio": "A golden retriever who is obsessed with fetch. Will retrieve anything, anytime, anywhere. Exceptionally friendly and loves children.",
      "name": "Buddy"
    },
    {
      "bio": "An old, wise beagle who has seen it all. Spends most of his day sleeping in a sunbeam, but occasionally perks up to share his wisdom (in the form of soulful howls).",
      "name": "Socrates"
    }
  ]
}

I'm OK with this.

[simonw]

I think this is good enough to ship as an alpha. I'll add minimal docs and do that.

[simonw]

Documentation:

• CLI: https://github.com/simonw/llm/blob/cb861cf29c4a691b6595332315cdea07fa6030a5/docs/usage.md#schemas
• Python API: https://github.com/simonw/llm/blob/cb861cf29c4a691b6595332315cdea07fa6030a5/docs/python-api.md#schemas
• Writing plugins: https://github.com/simonw/llm/blob/cb861cf29c4a691b6595332315cdea07fa6030a5/docs/plugins/advanced-model-plugins.md#supporting-schemas