Unable to use the newly launched Structured Outputs feature of OpenAI, response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ...} }

[ianva]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

 const TranslationSchema = z.object({
  translatedText: z.string(),
  entities: z.array(
    z.object({
      ner: z.string(),
      label: z.string(),
      translated: z.string().optional(),
    })
  ),
});
 ...
 
    return RunnableSequence.from([
      {
        sourceLanguage: () => formattingObject.sourceLanguage,
        targetLanguage: () => formattingObject.targetLanguage,
        genre: () => formattingObject.genre,
        textToTranslate: () => formattingObject.textToTranslate,
        untranslatedNerEntities: () => formattingObject.untranslatedNerEntities,
      },
      prompt,
      (promptValue: any) => {
        const boundModel = this.model.bind({
          response_format: zodResponseFormat(TranslationSchema, "translation"),
        });
        return boundModel.invoke(promptValue);
      },
    ]);

### Error Message and Stack Trace (if applicable)

_No response_

### Description

Unable to use the newly launched Structured Outputs feature of OpenAI, `response_format: { type: "json_schema", json_schema: {"strict": true, "schema": ...} }`

The type of response_format does not support json_schema, and the return value does not contain the JSON data defined by json_schema

### System Info

osx/bun

[dosubot]

Hey @ianva! I'm here to help you with any bugs, questions, or contributions you have for LangChain.js. Let's tackle this issue together while we wait for a human maintainer to join in.

To use the newly launched Structured Outputs feature of OpenAI with the response_format configuration in LangChain.js, you should use the withStructuredOutput method. This method allows you to create a model that returns output matching a JSON Schema or a Zod schema. Here is an example of how you can set it up:

import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
import { ChatOpenAI } from "@langchain/openai";
import { ChatPromptTemplate } from "@langchain/core/prompts";

// Define your Zod schema
const TranslationSchema = z.object({
  translatedText: z.string(),
  entities: z.array(
    z.object({
      ner: z.string(),
      label: z.string(),
      translated: z.string().optional(),
    })
  ),
});

// Create the structured output model
const model = new ChatOpenAI({ modelName: "gpt-4-turbo-preview", temperature: 0 });
const modelWithStructuredOutput = model.withStructuredOutput({
  schema: zodToJsonSchema(TranslationSchema),
  name: "translation",
});

const prompt = ChatPromptTemplate.fromMessages([
  "system",
  `You are a helpful assistant. Translate the following text and extract named entities.`,
  "human",
  "Translate this text: {textToTranslate}",
]);

const chain = prompt.pipe(modelWithStructuredOutput);
const result = await chain.invoke({
  textToTranslate: "Hello, world!",
});

console.log(result);

In this example:

1. You define a Zod schema (TranslationSchema) that specifies the structure of the expected output.
2. You create a structured output model using withStructuredOutput, passing in the Zod schema and additional options.
3. You create a prompt template and pipe it through the structured output model.
4. You call the chain with your input data to get the structured output.

This setup ensures that the output conforms to the specified JSON schema or Zod schema, leveraging the jsonMode method to validate and parse the output accordingly. This approach is more reliable for generating structured results, even with higher temperature values, making it suitable for more creative applications.

Note: If using jsonMode as the method, you must include context in your prompt about the structured output you want. This must include the keyword: JSON ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[ikalachy]

I bet the answer is not correct. It should be smth like
.withStructuredOutput(TranslationSchema, { name: "translate", });

And again with latest 3.1 update and tooling it seems to me functionality is broken somehow. I cant make model to generate appropriate content

[bracesproul]

Hey, we're implementing json_schema response format in #6438

[ianva]

Is the encapsulation of the ChatOpenAI class too complex, making it difficult to expand fields?
Could there be some configurations that allow for merging when new fields are added?

[bracesproul]

Updating the response_format type to allow

{
  type: "json_schema",
  json_schema: { ... }
}

is simple, however OpenAI added a new beta endpoint for making requests with this new json schema response format, as well as some utils for converting Zod schemas to their expected JSON schema format. Due to this, it's not as simple as expending the type, and actual code logic changes are required to make it work as intended.

[ianva]

Thanks for your work, looking forward to this update arriving soon

[david8z]

I am curious shouldn't withStructureOuput directly implement this whe calling OpenAI? cc/ @bracesproul

[bracesproul]

@david8z We can't make withStructuredOutput call with json_schema mode by default as that would technically be a breaking change (any change to default behavior = "breaking" change). I believe we are planning on making withStructuredOutput pass strict: true by default when we bump minor versions next.