Incomplete Data Returned from OpenAI API Model

[beiyanpiki]

Thanks for aweson work!

I have a question, when using the SmartScraperMultiGraph with the OpenAI API model (gpt-3.5-turbo), I need to return a List. However, due to the large amount of data, the information gets truncated, resulting in an incomplete List. Is there a way to continue requesting the API to return the complete information?

Code:

OPENAI_API_KEY = "xxxxxxxxxxxxxxxxxx"
graph_config = {
    "llm": {
        "api_key": OPENAI_API_KEY,
        "model": "gpt-3.5-turbo",
    },
    "verbose": True,
    "headless": False,
}

class MemberInfo(BaseModel):
    name: str = Field(description="The name of member")
    email: str = Field(description="The email of member")
    rank: str = Field(description="The rank of member")
    fields: List[str] = Field(description="The fields or research area of member")
    url: HttpUrl = Field(
        description="The personal homepage URL of the member, typically linked with their name")

class Group(BaseModel):
    list: List[MemberInfo] = Field(description="The list of member")


def getGroupMembers(urls: List[str]) -> List[str]:
    g = SmartScraperMultiGraph(
        prompt="""
     Please provide details for all members including: 
        - name: The name of member
        - email: The email of member
        - rank: The rank of member
        - fields: The fields or research area of member
        - url: The personal homepage URL of the member, typically linked with their name
     If any information is missing for a member, replace it with "NA" (leave fields empty for research areas). 
     URLs must be absolute, linking directly to their personal homepage.
     Ensure no member is skipped.
        """,
        source=urls,
        config=graph_config,
        schema=Group,
    )
    result = g.run()
    return result.get('list', [])

members = getGroupMembers(["https://example.com/a", "https://example.com/b"])

members

members
[
{'name': 'xxx', 'email': 'xxx', 'rank': 'xxx', 'fields': ['xxx'], 'url': 'https://example.com/xxx'},
...,
{'name': 'yyy', 'email': 'yyy', 'rank': 'yyy', 'fields': ['yyy'], 'url': 'https://example.com/yyy'},

{'name': 'zzz', 'email': 'zzz'} # <- Missing rank, fields and url, might be truncated.
]