Fix streaming error when using Azure OpenAI Service

[stevehodgkiss]

Trying out Azure OpenAI service and ran into an issue with streaming chat completions (gpt-4o version 2024-05-13).

A case clause error occurs in LLMChain.do_run (link). The issue is a message is being transformed and returning [].

16:20:35.537 [error] {{:case_clause, {:ok, [[], [%LangChain.MessageDelta{content: "", status: :incomplete, index: 0, role: :assistant, tool_calls: nil}], [%LangChain.MessageDelta{

The message from Azure OpenAI that results in a [] return value from do_process_return is (from here):

%{
  "choices" => [],
  "created" => 0,
  "id" => "",
  "model" => "",
  "object" => "",
  "prompt_filter_results" => [
    %{
      "content_filter_results" => %{
        "hate" => %{"filtered" => false, "severity" => "safe"},
        "self_harm" => %{"filtered" => false, "severity" => "safe"},
        "sexual" => %{"filtered" => false, "severity" => "safe"},
        "violence" => %{"filtered" => false, "severity" => "safe"}
      },
      "prompt_index" => 0
    }
  ]
}

I went with adding this to Utils.handle_stream_fn since it would prevent this error for other transform return values that potentially could return [], but an alternative fix would be to add the following to ChatOpenAI before L664:

  def do_process_response(_model, %{"choices" => []}) do
    :skip
  end

[brainlid]

Thanks!

❤️💛💙💜

[brainlid]

Instead of matching on the :skip or [], could a pattern match in the ChatOpenAI module match on the empty choices and return a :skip?

[stevehodgkiss]

Yep, that's an option. That's probably better now that I think about it a bit more, because having this in Utils won't prevent empty choices from slipping through as [] in the non-streaming case (although I didn't see this message/error in non-streaming). The reason I went with Utils is so that it would prevent this invalid state and subsequent error for any chat model, but again this would apply to the streaming case 🤔

Shall I open a PR to move this fix to ChatOpenAI?

[brainlid]

Shall I open a PR to move this fix to ChatOpenAI?

Yes please. Sorry, I thought more about the changed after I approved and merged it. Thanks!