[Core] Roadmap for handling context overflow

[yiranwu0]

Any help is appreciated!

Thit task demands a considerable amount of effort. If you have insights, suggestions, or can contribute in any way, your help would be immensely valued.

Problem Description

(Continue from #9) Current LLM have limited context size / token limit (gpt3.5turbo: 4096, gpt4 8192, etc). Although the current max_token limit from OpenAI is sufficient for many tasks, the token limit will be always exceeded with the conversation running. autogen.Completion will raise this InvalidRequestError that indicates the context size is exceeded since autogen doesn’t have a way to handle long context sizes.

Potential Methods

1. Compression: we can utilize LLMs to compress previous messages to reduce context size.
2. Retrieve related history messages: we can retrieve the most related messages based on the latest message.
3. Truncation: a simple way is keep the recent k messages and truncate all previous messages. We can also implement some truncation mechanisms, such as remove failed code executions.
4. A mixture of methods above.

Some References

• https://community.openai.com/t/when-conversation-grows-larger-did-chatgpt-include-all-of-the-conversation-context-for-every-chat/121876/7
• https://ai.stackexchange.com/questions/38150/how-does-chatgpt-retain-the-context-of-previous-questions
• From @juanmacuevas: MemWalker processed long context intro a tree of summaries and link to the paper
• LongLLMLingua: local model for compression.
• From @MrXandbadas: MemGPT supporting Autogen Agent discussion

Compression & Truncation

Preview Give feedback

1. Add compression agent #131
   feature long context handling +2
2. Add token_count_util #421
   +0
   
3. Add CompressibleAgent #443
   long context handling +1
4. Allow async compression
5. [Core] Compression in GroupChat #497
   group chat/teams long context handling +2
6. Add new mode "WINDOW" for compressible agent #685
   long context handling +1

Retrieval

Preview Give feedback

1. Explore memGPT agent

[OrderAndCh4oS]

Probably a slight tangent, but I'm finding error stack traces to a major culprit of context overflow

[kazunator]

Would using something like Llama Index help?

[kazunator]

could this work? https://memgpt.ai/

[juanmacuevas]

MemWalker processed long context intro a tree of summaries
Could this approach be applied for autogen? link to the paper

[sonichi]

@Hacker0912 Are you interested in this topic?

[qidanrui]

AutoGen is a great project! I'm very interested in how do you solve the context overflow.
When I was using AutoGen, sometimes I meet this situation:

Do you have any possible solution now? Thanks! @kevin666aa

[yiranwu0]

AutoGen is a great project! I'm very interested in how do you solve the context overflow. When I was using AutoGen, sometimes I meet this situation: Do you have any possible solution now? Thanks! @kevin666aa

@qidanrui Here is a experimental PR for compression: #131 . It would be great if you can check it out and test it!

Just found a potential good solution for compression and I will look into this: https://arxiv.org/abs/2310.06839

[qidanrui]

Thanks for sharing! @kevin666aa I'm so interested in the AutoGen project!
I have two other more general questions: 1. what is the difference between AutoGen and ChatArena? 2. Can we customize our own tool/agent except python code writing like what we can do in LangChain?

[aaronstevenson408]

i also am having issue with using a mistral model , using textgen webui as the api host

openai.error.InvalidRequestError: This model maximum context length is 2048 tokens. However, your messages resulted in over 2165 tokens.

[MrXandbadas]

could this work? https://memgpt.ai/

letta-ai/letta#65 (comment)

Edit:
Its been done! Fix your context issues by including a new bot. For the most up to date comment click the link but to save time here:
Please note this implementation replaces the coder from an example found in the examples folder

import os
import autogen
import asyncio
from absl import app, flags

config_list = [
    {
        'model': 'gpt-4',
        'api_key': os.getenv('OPENAI_API_KEY'),
    },
]

MEMGPT = True
if not MEMGPT:
    llm_config = {"config_list": config_list, "seed": 42}
    user_proxy = autogen.UserProxyAgent(
       name="User_proxy",
       system_message="A human admin.",
       code_execution_config={"last_n_messages": 2, "work_dir": "groupchat"},
       human_input_mode="TERMINATE"
    )
    coder = autogen.AssistantAgent(
        name="Coder",
        llm_config=llm_config,
    )
    pm = autogen.AssistantAgent(
        name="Product_manager",
        system_message="Creative in software product ideas.",
        llm_config=llm_config,
    )

else:
    import memgpt.autogen.memgpt_agent as memgpt_autogen
    import memgpt.autogen.interface as autogen_interface 
    import memgpt.agent as agent
    import memgpt.system as system
    import memgpt.utils as utils
    import memgpt.presets as presets
    import memgpt.constants as constants
    import memgpt.personas.personas as personas
    import memgpt.humans.humans as humans
    from memgpt.persistence_manager import InMemoryStateManager, InMemoryStateManagerWithPreloadedArchivalMemory, InMemoryStateManagerWithFaiss
    
    llm_config = {"config_list": config_list, "seed": 42}
    user_proxy = autogen.UserProxyAgent(
       name="User_proxy",
       system_message="A human admin.",
       code_execution_config={"last_n_messages": 2, "work_dir": "groupchat"},
    )

    interface = autogen_interface.AutoGenInterface()
    persistence_manager = InMemoryStateManager()
    memgpt_agent = presets.use_preset(presets.DEFAULT, 'gpt-4', personas.get_persona_text(personas.DEFAULT), humans.get_human_text(humans.DEFAULT), interface, persistence_manager)

    # MemGPT coder
    coder = memgpt_autogen.MemGPTAgent(
        name="MemGPT_coder",
        agent=memgpt_agent,
    )

    # non-MemGPT PM
    pm = autogen.AssistantAgent(
        name="Product_manager",
        system_message="Creative in software product ideas.",
        llm_config=llm_config,
    )

groupchat = autogen.GroupChat(agents=[user_proxy, coder, pm], messages=[], max_round=12)
manager = autogen.GroupChatManager(groupchat=groupchat, llm_config=llm_config)

user_proxy.initiate_chat(manager, message="First send the message 'Let's go Mario!'")

[PriNova]

I suggest starting with the simplest implementations without external dependencies first and later building on these.
This means to use truncation so that the history has a sliding window set by users if custom models will be used.

Later on, the user can switch between different implementations how to handle context window through truncation, vector embeddings DB or MemGPT.

[yiranwu0]

Hello, @MrXandbadas Thanks for the heads up! It's great to have a memGPT agent in AutoGen. It requires much effort to add memgpt, but it seems that we can work with people from memGPT to make it happen. I guess the first step would be to make memgpt a built-in agent in AutoGen. Then we can think about how users can switch with different options for context overflow.

@PriNova Thanks for the advice! Your suggestion of truncating history is brought up here #195.
I am making a PR to add a compression agent for the user to choose here #131.
There are two parts to this PR:

1. A implementation where user can switch between two modes at a pre-set token limit:
   TERMINATE: terminate the process before RateLimitError is raised; "COMPRESS": call a compression agent to compress previous messages.
2. A compression agent that does compression.

From the first part, it would be easy to add different ways to handle context later.
The compression agent is experimental but serves as a choice for now.

Please take a look at it if you are interested! @PriNova @MrXandbadas

[sonichi]

Sounds good, except "the first step would be to make memgpt a built-in agent in AutoGen." Before we make it a built-in agent, it'll be helpful to demonstrate one good use case of memgpt-based agent in autogen.

[JonMike12341234]

I must be totally misunderstanding the request for "one good use case". Would this approach not give AutoGen Agents near limitless long term memory storage?

[sonichi]

Just suggesting doing it step by step. Test-driven development.

[MrXandbadas]

Just suggesting doing it step by step. Test-driven development.

I completely agree. Testing it in an more robust setup would garner more fruitful insight as to if it was going to be beneficial for the system or just a hinderance in the continual prompt generation that allows these agents to function so flawlessly.

[ddwinhzy]

I think it's gonna be a great start.

[SDcodehub]

I am interested in this topic, is anyone working on this? I can help can work together with others

[yiranwu0]

I am interested in this topic, is anyone working on this? I can help can work together with others

Hello @SDcodehub, thanks for your interest! Currently I am working on adding a compressible agent #443. It could be used as an interface for different types of compression and truncations.
I think it is a good start pointing to know what we are doing.
I am starting a draft for compressible groupchat in #497.
The next step will be to allow async management of history.

On the other hand, it is also possible to utilize existing framework like memGPT. memGPT is actively supporting autogen agents. As @sonichi pointed out, we need to "demonstrate one good use case of memgpt-based agent in autogen". It is not hard to add a memGPT agent, but how to modify it to serve as a group memory requires more effort and thinking.

[Grigorij-Dudnik]

Hey guys!

The lack of possibility to see exact prompt which agent gets to a context and lack of ability of managing agent's contexts are the main problems of autogen in my opinion. Main reason for it is not even overflow of context window - the main reasons is the LLMs working much worse when had too long context with a lot of useless noise, and also generate unnecesary costs.

Maybe you remember, lack of input prompt visibility was big problem with Langchain, until they did a Langsmith. With autogen we have same problem again. In my opinion, we can't talk about any serious AI development if we can't see and edit input prompt of LLMs.

What do you think about it? Is such features of editing context (as summarizing or removing old messages) will be available in near future? Maybe there are already solutions exist I don't know about?

Cheers!

[sonichi]

I'm quite excited at #1091 by @rickyloynd-microsoft. It makes the teachability a composable capability to any conversable agent. More generally, the same mechanism may be used for solving other longstanding issues like long context handling and allowing other interesting capabilities to be defined. I like the extensibility and the composability of this approach. Reviews are welcome.
cc @kevin666aa

[sgjohnson1981]

I'm quite excited at #1091 by @rickyloynd-microsoft. It makes the teachability a composable capability to any conversable agent. More generally, the same mechanism may be used for solving other longstanding issues like long context handling and allowing other interesting capabilities to be defined. I like the extensibility and the composability of this approach. Reviews are welcome. cc @kevin666aa

@sonichi
So does @rickyloynd-microsoft 's PR add a feature that eliminates the need for MemGPT or other context length solutions? I can't tell from "...mechanism may be used..." Is this mechanism used now for context length, or will that be implemented later?

[rickyloynd-microsoft]

I'm quite excited at #1091 by @rickyloynd-microsoft. It makes the teachability a composable capability to any conversable agent. More generally, the same mechanism may be used for solving other longstanding issues like long context handling and allowing other interesting capabilities to be defined. I like the extensibility and the composability of this approach. Reviews are welcome. cc @kevin666aa

@sonichi So does @rickyloynd-microsoft 's PR add a feature that eliminates the need for MemGPT or other context length solutions? I can't tell from "...mechanism may be used..." Is this mechanism used now for context length, or will that be implemented later?

Teachability is just one capability added through this new mechanism, and teachability is not designed to compress context or memorize general things like MemGPT. But other capabilities (like MemGPT or other ways of handling long context) could be added through this general capability-addition mechanism.

[ekzhu]

Shall we close this issue as several recent PRs related to long context handling have merged. @kevin666aa

[yiranwu0]

Yes, thanks!

[JingPush]

Hi, I'm working on conversable agent flow with autogen. and really wants to know the status of handling context window length and truncate chat history.

I read the above conversation and have a few questions?

1. Is the status already completed and released?
2. Is there any docs or paper from autogen explaining how this is being handled specifically?

It would be really helpful if you can answer the question!

[ekzhu]

@JingPush current we use this: https://microsoft.github.io/autogen/docs/topics/long_contexts