Being more lax with API responses to allow for better compatibility with alternative services? #266
Description
There are various services offering an OpenAI-compatible API nowadays. Some are doing things a bit differently, which makes async-openai choke on their responses.
I've managed to reproduce the problem on TogetherAI (frequently down though) and OpenRouter. The latter seems more reliable and possibly provides some models that can be used for free, so it's probably easier to test against.
On successful chat completion generation, OpenRouter returns an API response that looks like this:
{
"id": "gen-abcd",
"model": "model-here",
"object": "chat.completion",
"created": 1725709829,
"choices": [
{
"logprobs": null,
"finish_reason": "eos",
"index": 0,
"message": {
"role": "assistant",
"content": "Hello there! How can I help you today? If you have any questions, I'm here to assist.",
"refusal": ""
}
}
],
"usage": {
"prompt_tokens": 26,
"completion_tokens": 24,
"total_tokens": 50
}
}.. which async-openai chokes on with:
failed to deserialize api response: unknown variant
eos, expected one ofstop,length,tool_calls,content_filter,function_callat line 51 column 180
Since I'm building a tool which aims to support both OpenAI and other OpenAI-compatible API endpoints, I either need to use both async-openai and another (more lax) library.. Or I need to open issues for all OpenAI-compatible providers and have them try to mimic OpenAI's API responses better (e.g. avoid using finish_reason = eos). Here's an instance where some folks fixed their API-compatibility to satisfy async-openai, but.. persuading various providers to do the same seems like a difficult endeavour.
It would probably be easier if async-openai was not so strict sometimes. This possibly goes in other directions as well - not taking pre-defined enums for certain parameters and letting people use strings instead (perhaps via an SomeEnum::Custom(String) variant).
I'm not sure if it's the goal of this library to allow working with OpenAI-compatible APIs or if it's something that you're against.