create_partial streaming not behaving as expected

[jakevollkommer]

• This is actually a bug report.
• I am not getting good LLM Results
• I have tried asking for help in the community on discord or discussions and have not received a response.
• I have tried searching the documentation and have not found an answer.

What Model are you using?

• gpt-3.5-turbo
• gpt-4-turbo
• gpt-4
• Other (gpt-4o)

Describe the bug
Partial streaming returns an extraction on every generation for the extraction stream, however fields are None on every generation until the final token, which is when it fully fills out. Essentially defeating the entire purpose of streaming.

To Reproduce

from pydantic import BaseModel, Field
from instructor import from_openai
from openai import OpenAI

class Summary(BaseModel):
    summary: str = Field(description="A detailed summary")

def main():
    client = from_openai(OpenAI())
    extraction_stream = client.chat.completions.create_partial(
        model="gpt-4o",
        response_model=Summary,
        messages=[
             {"role": "system", "content": "You summarize text"},
            {"role": "user", "content": " Summarize: Mary had a little lamb"}
        ],
        stream=True,
    )

    for extraction in extraction_stream:
        print(extraction)

main()

Expected behavior
This used to add new content on every token output, but now fields are always None until the last token for that field completes

Screenshots

[jxnl]

I was able to reproduce this but I don't really have time this week to take a look.

[jakevollkommer]

@jxnl not sure I'll have time before you but if I can dive in any quick tips on where to look?

[jxnl]

import pydantic_core

print(
    pydantic_core.from_json(
        '{"name": "jaso',
        allow_partial=True,
    )
)
# > {}


print(
    pydantic_core.from_json(
        '{"name": "jaso"',
        allow_partial=True,
    )
)
# > {'name': 'jaso'}

[jxnl]

pydantic/pydantic-core#1293

[jakevollkommer]

@jxnl i see, so does the partial stream have unclosed json up until the last token? Maybe we have to downgrade pydantic

[jakevollkommer]

@jxnl looks like pydantic/pydantic-core#1293 was closed, can you check if it solved the issue with partial streaming?

[jakevollkommer]

pydantic/jiter#101

[antondkg]

https://github.com/pydantic/jiter/releases/tag/v0.4.0

fix released 18hrs ago

[jakevollkommer]

Need to use jitter.from_json instead of pydantic_core.from_json

[jxnl]

ah! can you make a PR for this?

[jakevollkommer]

Haven't had a chance yet, might be able to this week

[jxnl]

Need to use jitter.from_json instead of pydantic_core.from_json

@ellipsis-dev can you try to make this change? and make sure imports are correct?

[devorboy]

@jxnl - I love the instructor package; simply indispensable.
Partial/steaming is no longer working with OpenAI, right?
Still generating None until the full answer is ready.

[jxnl]

@ivanleomk can you take a look at this?

[ivanleomk]

I tested it out with jitter and it seems to work! Will try working on a PR for this

from jiter import from_json
from pydantic_core import from_json as from_json_pydantic

partial_json = '{"name": "jaso'
parsed_data = from_json(partial_json.encode(), partial_mode="trailing-strings")
print("(Jitter)   : Parsed partial JSON:", parsed_data)
parsed_data = from_json_pydantic(partial_json.encode(), allow_partial=True)
print("(Pydantic) : Parsed partial JSON:", parsed_data)

This gives the output below

(Jitter)   : Parsed partial JSON: {'name': 'jaso'}
(Pydantic) : Parsed partial JSON: {}

[ivanleomk]

Closing this since #745 addresses this issue

[devorboy]

Great, it now works perfectly with OpenAI and Anthropic