How to configure the custom endpoint for azure openai service? (APIM endpoint with key)

[ronakshirolawala]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { AzureChatOpenAI } from "@langchain/openai";

export const model = new AzureChatOpenAI({
	 azureOpenAIBasePath:
		"APIM endpoint",
	headers: {
		"Api-Key": "key",
	},
});

Description

We have developed a chatbot that communicates with a WordPress application, and we've configured the solution to be web-based, keeping all configurations on the client side as we know this is not a good idea. However, we are facing an issue with the completion URL, which exposes our endpoint along with the deployment name, potentially making it vulnerable to unauthorized access. To mitigate this risk, we moved the OpenAI endpoint behind Azure API Management (APIM) to gain more control. Now, we want to call the endpoint directly without relying on client-side configuration, but we haven't found a supported way to do this. How can we achieve this?

As you can see in the image, completion call will be happening in each call and it consist of ApiKey,InstanceName,DeploymentName,ApiVersion but I just want to directly call the apim along with key.

System Info

"@langchain/azure-openai": "^0.0.11",
"@langchain/langgraph": "^0.2.24",
"langchain": "^0.3.6"

[dosubot[bot]]

Hello @ronakshirolawala! I'm here to help you with any bugs, questions, or contributions you have. Let me know how I can assist you!

I found a similar closed issue that might be relevant to your question: How to provide custom endpoint for Azure OpenAi service? (gateway URL). This issue was resolved by a pull request that added support for custom endpoints for Azure OpenAI ^[1].

To configure a custom endpoint for Azure OpenAI in LangChainJS, you can use the getEndpoint function with the appropriate configuration parameters. Here's an example of how you can set it up:

import { AzureChatOpenAI } from "@langchain/openai";

const model = new AzureChatOpenAI({
  azureOpenAIBasePath: "your-APIM-endpoint",
  azureOpenAIApiKey: "your-api-key",
  azureOpenAIApiInstanceName: "your-instance-name",
  azureOpenAIApiDeploymentName: "your-deployment-name",
  azureOpenAIApiVersion: "your-api-version",
  headers: {
    "Api-Key": "your-api-key",
  },
});

This setup allows you to specify a custom base path for your Azure OpenAI service, which can be your APIM endpoint. Make sure to replace the placeholders with your actual configuration values. This way, you can call the endpoint directly without exposing sensitive information on the client side ^[2][3].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To address the CORS issue you're encountering with LangChainJS, you can try setting the request mode to 'no-cors'. This can be done by configuring the request mode in the fetchAdapter function. Here's an example of how you can set it up:

const config = {
  url: 'https://www.plink.com/ai/openai/deployments/openai-link/chat/completions?api-version=2024-08-01-preview',
  method: 'POST',
  mode: 'no-cors', // Set the request mode to 'no-cors'
  headers: {
    'Api-Key': 'your-api-key',
    // other headers...
  },
  // other configurations...
};

fetchAdapter(config)
  .then(response => {
    // handle response
  })
  .catch(error => {
    // handle error
  });

This configuration sets the request mode to 'no-cors', which can help bypass CORS restrictions. However, keep in mind that using 'no-cors' mode limits the ability to read the response data. If you need to access the response, you might need to handle CORS on the server side by ensuring the server includes the Access-Control-Allow-Origin header in its responses ^[1].

To continue talking to Dosu, mention @dosu.

[ronakshirolawala]

Yes @dosu, you are right, but how can I configure the fetchAdapter in AzureChatOpenAI? as I am using the earlier mentioned code only!

const model = new AzureChatOpenAI({
azureOpenAIBasePath: "your-APIM-endpoint",
azureOpenAIApiKey: "your-api-key",
azureOpenAIApiInstanceName: "your-instance-name",
azureOpenAIApiDeploymentName: "your-deployment-name",
azureOpenAIApiVersion: "your-api-version",
headers: {
"Api-Key": "your-api-key",
},
});

[dosubot[bot]]

To configure a custom fetchAdapter in the AzureChatOpenAI class, you would need to extend the class and override the methods where HTTP requests are made. This allows you to implement your custom logic for handling HTTP requests. Here's an example of how you can achieve this:

import { AzureChatOpenAI } from "@langchain/openai";

class CustomAzureChatOpenAI extends AzureChatOpenAI {
  constructor(fields) {
    super(fields);
    // Additional initialization if needed
  }

  async _streamChatCompletionsWithRetry(azureOpenAIMessages, options) {
    // Implement custom HTTP request logic here using your custom fetchAdapter
    const customResponse = await customFetchAdapter({
      url: this.azureOpenAIBasePath,
      method: 'POST',
      mode: 'no-cors', // Set the request mode to 'no-cors'
      headers: {
        'Api-Key': this.azureOpenAIApiKey,
        // other headers...
      },
      // other configurations...
    });
    return customResponse;
  }

  async _generate(messages, options, runManager) {
    // Implement custom HTTP request logic here using your custom fetchAdapter
    const customResponse = await customFetchAdapter({
      url: this.azureOpenAIBasePath,
      method: 'POST',
      mode: 'no-cors', // Set the request mode to 'no-cors'
      headers: {
        'Api-Key': this.azureOpenAIApiKey,
        // other headers...
      },
      // other configurations...
    });
    return customResponse;
  }
}

// Usage
const model = new CustomAzureChatOpenAI({
  azureOpenAIBasePath: "your-APIM-endpoint",
  azureOpenAIApiKey: "your-api-key",
  azureOpenAIApiInstanceName: "your-instance-name",
  azureOpenAIApiDeploymentName: "your-deployment-name",
  azureOpenAIApiVersion: "your-api-version",
  headers: {
    "Api-Key": "your-api-key",
  },
});

In this example, customFetchAdapter would be your custom function that handles HTTP requests. You can set the request mode to 'no-cors' or any other configuration you need. This approach allows you to fully control how requests are made and handle CORS issues accordingly ^[1].

To continue talking to Dosu, mention @dosu.

[ronakshirolawala]

@dosu, I have tried with the suggestion still i am getting the same error of the cors.

[dosubot[bot]]

To address the CORS issue you're facing, you need to ensure that the server hosting your API endpoint includes the Access-Control-Allow-Origin header in its responses. This is a server-side configuration and cannot be fully resolved from the client side. Here are some steps you can take:

1. Server-Side Configuration: Ensure that your server is configured to include the Access-Control-Allow-Origin header in its responses. This header should allow requests from the origin where your application is hosted. For example, if your application is hosted at https://yourapp.com, the header should be set to Access-Control-Allow-Origin: https://yourapp.com. If you want to allow requests from any origin (not recommended for production), you can set it to *.

2. Preflight Requests: Make sure your server correctly handles preflight requests (OPTIONS requests) by including the necessary CORS headers in the response. This includes Access-Control-Allow-Methods and Access-Control-Allow-Headers.

3. Check Server Logs: Review your server logs to ensure that the CORS headers are being set correctly and that there are no errors when handling requests.

4. Local Testing: If you're testing locally, consider using a tool like a browser extension to temporarily disable CORS for testing purposes. However, this is not a solution for production environments.

If you have control over the server configuration, these changes should help resolve the CORS issue. If not, you may need to work with the server administrator to implement these changes.

To continue talking to Dosu, mention @dosu.