gpt-4o retrieved context difference in langchain, it changes, but stable for gpt-4 or gpt-3.5 models

[seroetr]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_core.messages import HumanMessage
from langchain_openai import AzureChatOpenAI

llm_s = AzureChatOpenAI(
    openai_api_version="2024-05-01-preview", 
    azure_deployment="gpt4o", 
    temperature=0,
    streaming=False, 
    #callbacks=[StreamingStdOutCallbackHandler()],
    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
    verbose=False,
    max_tokens=1000
)

llm2_s = AzureChatOpenAI(
    openai_api_version="2024-05-01-preview",
    azure_deployment="gpt4o", 
    temperature=0,
    streaming=False, 
    #callbacks=[StreamingStdOutCallbackHandler()],
    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
    verbose=False,
    max_tokens=5
)

from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-large")
#model = HuggingFaceCrossEncoder(model_name="cross-encoder/ms-marco-MiniLM-L-12-v2")
bge_compressor = CrossEncoderReranker(model=model, top_n=10)

bm25_retriever = BM25Retriever.from_documents(doc_chunks)
bm25_retriever.k = 10
ensemble_retriever = EnsembleRetriever(retrievers=[bm25_retriever, big_chunks_retriever], weights=[0.5,0.5])
compression_ensemble_retriever = ContextualCompressionRetriever(base_compressor=bge_compressor, base_retriever=ensemble_retriever)

memory3 = ConversationBufferWindowMemory(k=5, memory_key="chat_history", return_messages=True, output_key='answer',  input_key="question")

from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler

conversation_chain_sweden = ConversationalRetrievalChain.from_llm(
        llm = llm_s,
        retriever=compression_ensemble_retriever,
        memory = memory3,
        chain_type="stuff",
        combine_docs_chain_kwargs={'prompt': QA_CHAIN_PROMPT},
        condense_question_llm = llm2_s,
        #condense_question_llm = llm2_gpt35,
        #get_chat_history=lambda h :h,
        get_chat_history = _get_chat_history,
        return_generated_question=False,
        rephrase_question=False,
        return_source_documents=False,
        #verbose=False,
        verbose=True,
    )
    
    def soru_sor_sweden(question):
    with get_openai_callback() as cb:
        # Ensure the response here is structured as a dictionary with an "answer" key
        try:
            response=conversation_chain_sweden.invoke({"question": question, "chat_history": memory3})["answer"]
        
        except ValueError:
            response = ("Invalid Question")
        except:
            response= ("Invalid Question")
        #print(response["answer"])
    return response

Description

When I perform RAG in the relevant conversation chain, I have a fixed question, and when I use GPT-4 or GPT-3.5, the retrieved contexts within the prompt in debug mode contain the contexts that include the answer to my question. However, when I select the GPT-4o language model, the context containing the correct answer to the same question does not appear. In this case, the entire chain, code, and retriever remain the same; only the LLM model is changed. The context for RAG within the prompt changes when using GPT-4o.

System Info

absl-py 2.1.0
accelerate 0.29.0.dev0
adal 1.2.7
aiofiles 23.2.1
aiohttp 3.9.1
aiosignal 1.3.1
alembic 1.13.1
altair 5.2.0
annotated-types 0.6.0
antlr4-python3-runtime 4.9.3
anyio 4.3.0
appdirs 1.4.4
applicationinsights 0.11.10
argcomplete 3.2.2
asgiref 3.7.2
asttokens 2.0.5
async-timeout 4.0.3
attrs 23.2.0
audioread 3.0.1
azure-common 1.1.28
azure-core 1.29.7
azure-graphrbac 0.61.1
azure-identity 1.15.0
azure-mgmt-authorization 4.0.0
azure-mgmt-containerregistry 10.3.0
azure-mgmt-core 1.4.0
azure-mgmt-keyvault 10.3.0
azure-mgmt-network 25.1.0
azure-mgmt-resource 23.0.1
azure-mgmt-storage 21.1.0
azure-storage-blob 12.13.0
azureml-core 1.54.0
azureml-dataprep 5.1.3
azureml-dataprep-native 41.0.0
azureml-dataprep-rslex 2.22.2
azureml-fsspec 1.3.0
azureml-inference-server-http 0.4.14
azureml-mlflow 1.54.0
backoff 2.2.1
backports.tempfile 1.0
backports.weakref 1.0.post1
bcrypt 4.1.2
beautifulsoup4 4.12.3
bitarray 2.9.2
bitsandbytes 0.43.0
boto3 1.34.107
botocore 1.34.107
bs4 0.0.2
build 1.0.3
cachetools 5.3.2
catalogue 2.0.10
certifi 2024.2.2
cffi 1.16.0
chardet 5.2.0
charset-normalizer 3.3.2
chroma-hnswlib 0.7.3
chromadb 0.4.22
click 8.1.7
cloudpickle 2.2.1
cohere 5.5.0
colbert-ai 0.2.19
colorama 0.4.6
coloredlogs 15.0.1
colorlog 6.8.2
comm 0.2.1
contextlib2 21.6.0
contourpy 1.2.0
cryptography 41.0.7
curl_cffi 0.6.0b9
cycler 0.12.1
databricks-cli 0.18.0
dataclasses-json 0.6.4
datasets 2.16.1
debugpy 1.6.7
decorator 5.1.1
Deprecated 1.2.14
dill 0.3.7
dirtyjson 1.0.8
distro 1.9.0
docker 6.1.3
docker-pycreds 0.4.0
docstring-inheritance 2.1.2
docstring-parser 0.15
docx2txt 0.8
duckduckgo_search 4.4
effdet 0.4.1
emoji 2.10.0
entrypoints 0.4
et-xmlfile 1.1.0
evaluate 0.4.1
exceptiongroup 1.2.0
executing 0.8.3
faiss-cpu 1.7.4
faiss-gpu 1.7.2
fast-pytorch-kmeans 0.2.0.1
fastapi 0.109.0
fastavro 1.9.4
fastembed 0.3.1
ffmpy 0.3.1
filelock 3.13.1
filetype 1.2.0
FlashRank 0.2.0
Flask 1.0.3
flatbuffers 23.5.26
fonttools 4.47.2
frozenlist 1.4.1
fsspec 2023.10.0
git-python 1.0.3
gitdb 4.0.11
GitPython 3.1.41
google-api-core 2.15.0
google-auth 2.27.0
google-auth-oauthlib 1.2.0
google-cloud-vision 3.7.2
googleapis-common-protos 1.62.0
gradio 4.16.0
gradio_client 0.8.1
greenlet 3.0.3
grpcio 1.64.0
grpcio-status 1.62.2
grpcio-tools 1.60.0
gunicorn 20.1.0
h11 0.14.0
h2 4.1.0
hpack 4.0.0
httpcore 1.0.4
httptools 0.6.1
httpx 0.27.0
httpx-sse 0.4.0
huggingface-hub 0.23.4
humanfriendly 10.0
hyperframe 6.0.1
idna 3.6
importlib-metadata 6.11.0
importlib-resources 6.1.1
inference-schema 1.3.0
install 1.3.5
InstructorEmbedding 1.0.1
iopath 0.1.10
ipykernel 6.28.0
ipython 8.20.0
ipywidgets 8.1.1
isodate 0.6.1
itsdangerous 1.1.0
jedi 0.18.1
jeepney 0.8.0
Jinja2 3.0.3
jiwer 2.5.2
jmespath 1.0.1
joblib 1.3.2
jsonpatch 1.33
jsonpath-python 1.0.6
jsonpickle 3.0.2
jsonpointer 2.4
jsonschema 4.21.1
jsonschema-specifications 2023.12.1
jupyter_client 8.6.0
jupyter_core 5.5.0
jupyterlab-widgets 3.0.9
kiwisolver 1.4.5
knack 0.11.0
kubernetes 29.0.0
langchain 0.2.6
langchain-community 0.2.6
langchain-core 0.2.10
langchain-experimental 0.0.62
langchain-huggingface 0.0.3
langchain-openai 0.1.13
langchain-text-splitters 0.2.2
langchainhub 0.1.15
langdetect 1.0.9
langgraph 0.0.21
langsmith 0.1.82
lark 1.1.9
layoutparser 0.3.4
lazy_loader 0.3
librosa 0.10.1
llama-index 0.10.36
llama-index-agent-openai 0.2.4
llama-index-cli 0.1.12
llama-index-core 0.10.36
llama-index-embeddings-azure-openai 0.1.9
llama-index-embeddings-fastembed 0.1.4
llama-index-embeddings-huggingface 0.2.0
llama-index-embeddings-instructor 0.1.3
llama-index-embeddings-openai 0.1.9
llama-index-indices-managed-llama-cloud 0.1.6
llama-index-legacy 0.9.48
llama-index-llms-azure-openai 0.1.8
llama-index-llms-openai 0.1.18
llama-index-llms-replicate 0.1.3
llama-index-multi-modal-llms-openai 0.1.5
llama-index-program-openai 0.1.6
llama-index-question-gen-openai 0.1.3
llama-index-readers-file 0.1.22
llama-index-readers-llama-parse 0.1.4
llama-index-vector-stores-qdrant 0.2.8
llama-parse 0.4.2
llamaindex-py-client 0.1.19
llvmlite 0.41.1
loguru 0.7.2
loralib 0.1.2
lxml 5.1.0
Mako 1.3.0
Markdown 3.5.2
markdown-it-py 3.0.0
MarkupSafe 2.1.4
marshmallow 3.21.1
matplotlib 3.8.2
matplotlib-inline 0.1.6
mdurl 0.1.2
mlflow 2.4.1
mlflow-skinny 2.10.0
mmh3 4.1.0
monotonic 1.6
mpmath 1.3.0
msal 1.26.0
msal-extensions 1.0.0
msg-parser 1.2.0
msgpack 1.0.7
msrest 0.7.1
msrestazure 0.6.4
multidict 6.0.5
multiprocess 0.70.15
mypy-extensions 1.0.0
ndg-httpsclient 0.5.1
neo4j 5.18.0
nest-asyncio 1.6.0
networkx 3.2.1
ninja 1.11.1.1
nltk 3.8.1
numba 0.58.1
numpy 1.26.4
nvidia-cublas-cu11 11.10.3.66
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu11 8.5.0.96
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.18.1
nvidia-nvjitlink-cu12 12.3.101
nvidia-nvtx-cu12 12.1.105
oauthlib 3.2.2
olefile 0.47
omegaconf 2.3.0
onnx 1.15.0
onnxruntime 1.17.3
openai 1.35.7
opencensus 0.11.4
opencensus-context 0.1.3
opencensus-ext-azure 1.1.13
opencv-python 4.9.0.80
openpyxl 3.1.2
opentelemetry-api 1.22.0
opentelemetry-exporter-otlp-proto-common 1.22.0
opentelemetry-exporter-otlp-proto-grpc 1.22.0
opentelemetry-instrumentation 0.43b0
opentelemetry-instrumentation-asgi 0.43b0
opentelemetry-instrumentation-fastapi 0.43b0
opentelemetry-proto 1.22.0
opentelemetry-sdk 1.22.0
opentelemetry-semantic-conventions 0.43b0
opentelemetry-util-http 0.43b0
orjson 3.10.0
overrides 7.7.0
packaging 23.2
pandas 2.2.1
paramiko 3.4.0
parso 0.8.3
pathspec 0.12.1
pdf2image 1.17.0
pdfminer.six 20231228
pdfplumber 0.11.0
peft 0.10.0
pexpect 4.8.0
pikepdf 8.15.1
pillow 10.3.0
pillow_heif 0.16.0
pip 24.1.1
pkginfo 1.9.6
platformdirs 3.10.0
pooch 1.8.0
portalocker 2.8.2
posthog 3.3.3
prompt-toolkit 3.0.43
proto-plus 1.23.0
protobuf 4.23.4
psutil 5.8.0
ptyprocess 0.7.0
pulsar-client 3.4.0
pure-eval 0.2.2
pyarrow 12.0.1
pyarrow-hotfix 0.6
pyasn1 0.5.1
pyasn1-modules 0.3.0
pycocotools 2.0.7
pycparser 2.21
pydantic 2.6.4
pydantic_core 2.16.3
pydub 0.25.1
Pygments 2.15.1
PyJWT 2.8.0
PyMuPDF 1.23.26
PyMuPDFb 1.23.22
PyNaCl 1.5.0
pynvml 11.5.0
pyOpenSSL 23.3.0
pypandoc 1.13
pyparsing 3.1.1
pypdf 4.1.0
pypdfium2 4.30.0
PyPika 0.48.9
pyproject_hooks 1.0.0
PySocks 1.7.1
PyStemmer 2.2.0.1
pytesseract 0.3.10
python-dateutil 2.9.0.post0
python-docx 1.1.2
python-dotenv 1.0.1
python-iso639 2024.1.2
python-magic 0.4.27
python-multipart 0.0.6
python-pptx 0.6.23
pytz 2024.1
PyYAML 6.0.1
pyzmq 25.1.2
qdrant-client 1.7.2
querystring-parser 1.2.4
RAGatouille 0.0.8.post2
rank-bm25 0.2.2
rapidfuzz 2.13.7
referencing 0.33.0
regex 2023.12.25
requests 2.31.0
requests-mock 1.12.1
requests-oauthlib 1.3.1
responses 0.18.0
rich 13.7.0
rpds-py 0.17.1
rsa 4.9
ruff 0.1.14
s3transfer 0.10.1
safetensors 0.4.2
scikit-learn 1.4.0
scipy 1.12.0
SecretStorage 3.3.3
semantic-router 0.0.43
semantic-version 2.10.0
sentence-transformers 2.7.0
sentencepiece 0.1.99
sentry-sdk 1.42.0
setproctitle 1.3.3
setuptools 68.2.2
shellingham 1.5.4
shtab 1.6.5
six 1.16.0
smmap 5.0.1
sniffio 1.3.1
snowballstemmer 2.2.0
soundfile 0.12.1
soupsieve 2.5
soxr 0.3.7
SQLAlchemy 2.0.28
sqlparse 0.4.4
srsly 2.4.8
stack-data 0.2.0
starlette 0.35.1
striprtf 0.0.26
sympy 1.12
tabulate 0.9.0
tenacity 8.2.3
tensorboard 2.15.1
tensorboard-data-server 0.7.2
threadpoolctl 3.2.0
tiktoken 0.7.0
timm 1.0.3
tokenizers 0.19.1
tomli 2.0.1
tomlkit 0.12.0
toolz 0.12.1
torch 2.1.2
torchaudio 2.1.2
torchvision 0.16.2
tornado 6.3.3
tqdm 4.66.2
traitlets 5.7.1
transformers 4.40.2
triton 2.1.0
trl 0.7.10
typer 0.9.0
types-requests 2.31.0.20240406
typing_extensions 4.10.0
typing-inspect 0.9.0
tyro 0.7.0
tzdata 2024.1
ujson 5.9.0
unsloth 2024.3
unstructured 0.14.2
unstructured-client 0.16.0
unstructured-inference 0.7.33
unstructured.pytesseract 0.3.12
urllib3 2.2.1
uvicorn 0.27.0
uvloop 0.19.0
voyager 2.0.2
wandb 0.16.4
watchfiles 0.21.0
wcwidth 0.2.5
websocket-client 1.7.0
websockets 11.0.3
Werkzeug 1.0.1
wheel 0.43.0
widgetsnbextension 4.0.9
wrapt 1.16.0
xformers 0.0.25
xlrd 2.0.1
XlsxWriter 3.2.0
xmltodict 0.13.0
xxhash 3.4.1
yarl 1.9.4
yfiles_jupyter_graphs 1.7.2
zipp 3.17.0

[dosubot[bot]]

Hey there, @seroetr! I'm here to help you with any bugs, questions, or contributions you might have. Let's work together to solve this!

The issue you're experiencing with the GPT-4o model not retrieving the correct context in your LangChain RAG setup, while GPT-4 and GPT-3.5 do, might be due to the specific model configuration in the rag_chain function. The code explicitly sets the model to "gpt-4-1106-preview":

model = ChatOpenAI(temperature=0, model="gpt-4-1106-preview", max_tokens=1024)

If you are using a different model identifier for GPT-4o, it might not be correctly configured or supported in the same way as GPT-4 or GPT-3.5. Ensure that the model identifier for GPT-4o is correctly specified and supported by the ChatOpenAI class.

Additionally, ensure that your retriever and rag_chain are properly configured to handle the specific model you are using. For example, the retriever should be correctly set up to retrieve relevant documents, and the rag_chain should be properly constructed to format and pass the context to the model:

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Make sure that all components in the chain are compatible with the GPT-4o model and that the model is correctly integrated into the LangChain framework. Also, ensure you are using the latest versions of the required packages:

! pip install -U langchain openai chromadb langchain-experimental

This setup should help align your configuration with the advanced RAG evaluation guidelines provided in the LangChain documentation ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[seroetr]

@dosu, Is gpt-4o identifier for azure openai available? Didnt you integrate into langchain enviroment? Also, the answer is not helpful.