[Bug/Feature]: vertex_ai/meta/llama-3.2-90b-vision-instruct-maas isn't always outputting stuff correctly for streaming

[Manouchehri]

What happened?

Sometimes, the content field returned for vertex_ai/meta/llama-3.2-90b-vision-instruct-maas while streaming is invalid.

curl -v "${OPENAI_API_BASE}/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "llama-3.2-90b-vision-instruct",
    "max_tokens": 1000,
    "seed": 31337,
    "stream": true,
    "temperature": 1.0,
    "messages": [
      {
        "role": "user",
        "content": "Hello"
      }
    ]
  }'

data: {"id":"chatcmpl-0d8df450-b016-4864-afc9-a17574ce00a9","choices":[{"index":0,"delta":{"content":"[{'type': 'text', 'text': 'Hi! How can I","role":"assistant"}}],"created":1729528781,"model":"meta/llama-3.2-90b-vision-instruct-maas","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-0d8df450-b016-4864-afc9-a17574ce00a9","choices":[{"index":0,"delta":{"content":" assist you today?'}]"}}],"created":1729528781,"model":"meta/llama-3.2-90b-vision-instruct-maas","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-0d8df450-b016-4864-afc9-a17574ce00a9","choices":[{"finish_reason":"stop","index":0,"delta":{}}],"created":1729528781,"model":"meta/llama-3.2-90b-vision-instruct-maas","object":"chat.completion.chunk"}

data: {"id":"chatcmpl-0d8df450-b016-4864-afc9-a17574ce00a9","choices":[{"index":0,"delta":{}}],"created":1729528781,"model":"meta/llama-3.2-90b-vision-instruct-maas","object":"chat.completion.chunk","usage":{"completion_tokens":21,"prompt_tokens":1,"total_tokens":22}}

data: [DONE]

When I do the request directly to Vertex AI, I do see this bug sometimes. So it's not LiteLLM.

data: {"choices":[{"delta":{"content":" [{'type': 'text', 'text': 'Hi, how can I assist","role":"assistant"},"index":0,"logprobs":null}],"created":1729529043,"id":"2024-10-21|09:44:03.767932-07|7.229.174.251|-1508838996","model":"meta/llama-3.2-90b-vision-instruct-maas","object":"chat.completion.chunk","system_fingerprint":""}

data: {"choices":[{"delta":{"content":" you today?'}]","role":"assistant"},"index":0,"logprobs":null}],"created":1729529043,"id":"2024-10-21|09:44:03.767932-07|7.229.174.251|-1508838996","model":"meta/llama-3.2-90b-vision-instruct-maas","object":"chat.completion.chunk","system_fingerprint":""}

data: {"choices":[{"delta":{"content":"","role":"assistant"},"finish_reason":"stop","index":0,"logprobs":null}],"created":1729529043,"id":"2024-10-21|09:44:03.767932-07|7.229.174.251|-1508838996","model":"meta/llama-3.2-90b-vision-instruct-maas","object":"chat.completion.chunk","system_fingerprint":"","usage":{"completion_tokens":21,"prompt_tokens":1,"total_tokens":22}}

data: [DONE]

I would really like if #5416 became a solution, so we could selectively disable streaming for buggy endpoints ourselves.

Relevant log output

No response

Twitter / LinkedIn details

https://www.linkedin.com/in/davidmanouchehri/

[ishaan-jaff]

hi @Manouchehri this is not a LiteLLM bug and sounds like a duplicate of #5416 - can you close this issue and add this as a comment there