Question related to InMemoryCache

[nithinnair23]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.cache import InMemoryCache
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-3.5-turbo-0125")
set_llm_cache(InMemoryCache())

# The first time, it is not yet in cache, so it should take longer
llm.predict("Tell me a joke")

Description

I wanted to know how much space/memory can be consumed for caching my asked questions, i.e what is the upper limit on this caching in terms of memory. Is it the entire memory or memory being allocated to Langchain application that is running on device?

System Info

OS: Windows 10
Python: 3.12
Langchain: 0.1.16

[dosubot[bot]]

The upper limit on memory consumption for caching questions using InMemoryCache in LangChain is 2000 tokens ^[1].

To continue talking to Dosu, mention @dosu.

[nithinnair23]

Hello,
The source cited by you is related to conversational history limit. I am asking about "llm caching" using InMemoryCache method. What is the upper limit on this caching in terms of memory. Is it the entire memory or memory being allocated to Langchain application that is running on device?

[dosubot[bot]]

The InMemoryCache method in LangChain does not have a configurable upper limit on memory consumption. It uses an in-memory dictionary to store cached items, and there are no built-in parameters or methods to set a memory limit. Therefore, the memory consumption is only limited by the available memory on the device running the LangChain application ^[1].

To continue talking to Dosu, mention @dosu.

[rambihari]

@dosu To follow up on what you mentioned above, you seem to suggest/imply that the InMemoryCache actually has no eviction strategy and is solely limited by available memory. So when the InMemoryCache hits the allotted memory limit how does it evict/does it just throw an OOM error?