How to allow max-token 8192 for Anthropic claude sonet 3.5?

[sessycode]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

       model: "claude-3-5-sonnet-20240620",
      temperature: 0,
      maxTokens: 8192, /// how to allow 8192 > 4096 ?
    });```      

### Error Message and Stack Trace (if applicable)

BadRequestError: 400 {"type":"error","error":{"type":"invalid_request_error","message":"max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up to 8192 output tokens, specify the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header."}}
    at Function.generate (Project\node_modules\.pnpm\@anthropic-ai+sdk@0.22.0\node_modules\@anthropic-ai\sdk\src\error.ts:58:14)
    at Anthropic.makeStatusError (Project\node_modules\.pnpm\@anthropic-ai+sdk@0.22.0\node_modules\@anthropic-ai\sdk\src\core.ts:383:21)
    at Anthropic.makeRequest (Project\node_modules\.pnpm\@anthropic-ai+sdk@0.22.0\node_modules\@anthropic-ai\sdk\src\core.ts:446:24)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async RetryOperation._fn (Project\node_modules\.pnpm\p-retry@4.6.2\node_modules\p-retry\index.js:50:12) {
  status: 400,
  headers: {
    'cf-cache-status': 'DYNAMIC',
    'cf-ray': '8ab5b4830e961f33-NRT',
    connection: 'keep-alive',
    'content-length': '286',
    'content-type': 'application/json',
    date: 'Tue, 30 Jul 2024 13:30:45 GMT',
    'request-id': 'hidden',
    server: 'cloudflare',
    via: '1.1 google',
    'x-cloud-trace-context': 'hidden',
    'x-should-retry': 'false'
  },
  error: {
    type: 'error',
    error: {
      type: 'invalid_request_error',
      message: "max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up to 8192 output tokens, specify the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header."
    }
  },
    error: {
      type: 'invalid_request_error',
      message: "max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up to 8192 output tokens, specify the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header."
    }
    error: {
      type: 'invalid_request_error',
      message: "max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up to 8192 output tokens, specify the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header."
    error: {
      type: 'invalid_request_error',
      message: "max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up to 8192 output tokens, specify the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header."
    error: {
      type: 'invalid_request_error',
      message: "max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up t    error: {
      type: 'invalid_request_error',
    error: {
    error: {
      type: 'invalid_request_error',
      message: "max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up to 8192 output tokens, specify the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header."
    }
  },
  attemptNumber: 1,
  retriesLeft: 6
}

Node.js v20.10.0

### Description

I have a prompt to generate long enough contents using sonet 3.5.
But it seems not allow the max-token(8192) which is officially allowed in Claude.
I have error message saying "max_tokens: 8192 > 4096, which is the maximum allowed number of output tokens for claude-3-5-sonnet-20240620. To access up to 8192 output tokens, specify the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header."

### System Info

pnpm package:
{
  "name": "ts-boilerplate",
  "version": "1.0.0",
  "description": "",
  "type": "module",
  "main": "index.js",
  "scripts": {
    "lint": "eslint . --ext .ts",
    "lint:fix": "eslint . --ext .ts --fix",
    "test": "tsx ./src/test/test_main.ts"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "@types/node": "^20.14.9",
    "@typescript-eslint/eslint-plugin": "^7.15.0",
    "@typescript-eslint/parser": "^7.15.0",
    "dotenv": "^16.4.5",
    "eslint": "^8.57.0",
    "eslint-config-prettier": "^9.1.0",
    "prettier": "^3.3.2",
    "tsx": "^4.16.2",
    "typescript": "^5.5.3"
  },
  "dependencies": {
    "@aws-sdk/client-bedrock-runtime": "^3.609.0",
    "@langchain/anthropic": "^0.2.3",
    "@langchain/community": "^0.2.17",
    "@langchain/core": "0.2.14",
    "@langchain/google-genai": "^0.0.21",
    "@langchain/google-vertexai": "^0.0.19",
    "@langchain/langgraph": "^0.0.26",
    "@langchain/openai": "^0.2.1",
    "langchain": "0.2.8",
    "zod": "^3.23.8",
    "zod-to-json-schema": "^3.23.1"
  },
  "pnpm": {
    "overrides": {
      "@langchain/core": "0.2.14"
    }
  }
}

windows 11
Node.js v20.10.0
pnpm version: 8.8.0

[dosubot]

To allow a max-token limit of 8192 for Anthropic Claude Sonnet 3.5 in LangChain.js, you need to set the maxTokensToSample parameter to 8192 and include the 'anthropic-beta: max-tokens-3-5-sonnet-2024-07-15' header in the clientOptions. Here is the modified code:

import { ChatAnthropic } from "@langchain/anthropic";

const model = new ChatAnthropic({
  model: "claude-3-5-sonnet-20240620",
  temperature: 0,
  maxTokensToSample: 8192,
  clientOptions: {
    defaultHeaders: {
      "X-Api-Key": process.env.ANTHROPIC_API_KEY,
      "anthropic-beta": "max-tokens-3-5-sonnet-2024-07-15"
    },
  },
});

const res = await model.invoke("Why is the sky blue?");

console.log(res);

This configuration sets the maxTokensToSample parameter to 8192 and includes the required custom header to allow the model to handle up to 8192 tokens ^[1][2].

To continue talking to Dosu, mention @dosu.

[bracesproul]

Looks like we need to allow for custom headers, I'll put up a PR sometime today adding this. Thanks for flagging!