LangChain and LlamaIndex support

[ktolias]

Excellent job, it made my LLM blazing fast. I tried it on T4 (16GB vRAM) and it seems to lower inference time from 36 secs to just 9 secs.

I then tried to use it along with LangChain and LlamaIndex but I got the following error:

ValidationError: 1 validation error for LLMChain
llm
value is not a valid dict (type=type_error.dict)

Can you please provide any guidance?

[beratcmn]

Langchain integration is easier than it looks. You can add vllm.LLM as a custom LLM. Doc link

[ktolias]

Even though my custom LLM responses fast using vLLM, when I hook it on Langchain or LlamaIndex it just hanging.

Any ideas?

CustomLLM

Inference with CustomLLM using vLLM

Inference with Langchain (keeps hanging)

Inference with LlamaIndex (keeps hanging)

[ktolias]

It seems that there is a problem with the specific chain (RetrievalQA).
When I revert to the simple LLMChain, everything worked fine with LangChain.
The problem with LlamaIndex still remains.

[zhuohan123]

Please refer to https://python.langchain.com/docs/integrations/llms/vllm for the latest langchain integration!