Use with langchain?

[frankandrobot]

How can vllm be used with langchain?

• since it produces an OpenAPI compatible API, the first idea that I had was to spin up a vvlm server and then use langchain's OpenApi class and point it at that instance
• but ....
• is there a simpler way? Ex: is it possible to use the LLM class directly with langchain? If so, is there an example?

[mspronesti]

Hi, have a look at this notebook.
Hope this helps.

[mspronesti]

Hi again @frankandrobot. Besides the notebook linked above, you can check the one that I recently added to langchain's docs (here).

In addition, I opened a PR to include some instructions on how to use vLLM with Langchain.
Hope this helps :) If it does, please mark this as an answer.

[pvtoan]

Hi @mspronesti , does this LangChain-VLLM support quantized model?

Because the vllm-project already supported quantized model (AWQ format) as shown in #1032

However, when I use the same way and just pass "quantization='awq" to your LangChain-VLLM, it seems does not work and just show OOM.

model_path = "/home/quadrep/toan/projects/LLMs/weights/vicuna-33B-AWQ"
model = VLLM(model=model_path, tensor_parallel_size=2, trust_remote_code=True, max_new_tokens=512, quantization='awq')
--> Error: torch.cuda.OutOfMemoryError: CUDA out of memory.

[hymie122]

I met the same problem

[pvtoan]

Hi,

@mspronesti answered me here #1162 (comment)

It works well.

[hymie122]

Thanks!

[mspronesti]

The PR has been merged. I suppose we can mark this discussion as completed :)

[tonyaw]

Hello, I'm using https://python.langchain.com/docs/integrations/llms/vllm#openai-compatible-server to communicate a vllm server.
How to set openai_api_key?
My code:

llm = VLLMOpenAI(
    openai_api_key="EMPTY",
    openai_api_base="https://vllm_server_url/v1",
    model_name="mistralai/Mixtral-8x7B-Instruct-v0.1",
    model_kwargs={"stop": ["."]},
)

I got following error:

  File "/home/worker/app/.venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for TavilySearchAPIWrapper
__root__
  Did not find tavily_api_key, please add an environment variable `TAVILY_API_KEY` which contains it, or pass `tavily_api_key` as a named parameter. (type=value_error)

[tonyaw]

I think I got it. TAVILY_API_KEY is for
tools = [TavilySearchResults(max_results=1)]
The next question is, my vllm server doesn't set "api_key". Any way to also not set api_key from langchain side?