Discussions

• 1 You must be logged in to vote

  🙏

  Mistral-7B model output is streamed on the remote server but not on the client side

  Alf-Z-SymphoMe asked May 26, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Not able to start server with swap mode

  sathyanarays asked May 25, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  is there a way to terminal model server?

  gabohouhou asked May 24, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  Unable to load mistralai/Mistral-7B-Instruct-v0.2. repository not found

  gningue asked Apr 27, 2024 in Q&A · Unanswered

  

  

  

  

  3
• 9 You must be logged in to vote

  🙏

  Total number of attention heads (X) must be divisible by tensor parallel size (Y).

  tamastarjanyi asked Sep 14, 2023 in Q&A · Unanswered

  

  

  

  

  3
• 2 You must be logged in to vote

  🙏

  How do i increase inference speed for a given model?

  tsvisab asked May 17, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  How do I find / build the Neuron docker image?

  yaronr asked Apr 26, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Profiling llama on AMD backend, found long times CopyHostToDevice

  CrimsonDump asked May 16, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  Mistral-7B-Instruct-v0.2 inference slows down when implemented on the offline_inference.py example, possibly due to improper usage

  Alf-Z-SymphoMe started May 14, 2024 in General · Closed

  

  

  0
• 2 You must be logged in to vote

  💬

  Regression in support of customized "role" in OpenAI compatible API (v.0.4.2)

  tanliboy started May 10, 2024 in General

  

  

  0
• 1 You must be logged in to vote

  🙏

  QUESTION REGARDING VRAM USAGE WHEN LOADING THE MODEL

  Fahmie23 asked May 10, 2024 in Q&A · Unanswered

  

  

  0
• 10 You must be logged in to vote

  🙏

  How to best set the --max-num-batched-tokens arg?

  edwardzjl asked May 10, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  [Bug]: 使用vllm 部署用着用着 模型突然没有响应了 nvidia-smi多了个execute method

  lxb0425 asked May 8, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  what is the difference between litellm and vllm?

  YunchaoYang started Apr 26, 2024 in General

  

  

  2
• 3 You must be logged in to vote

  🙏

  Question regarding the nearly double GPU memory consumption.

  Zhuqln asked Jun 25, 2023 in Q&A · Answered

  

  

  

  

  8
• 1 You must be logged in to vote

  🙌

  LLM Controller Interface for vLLM (LLM output format controller)

  Ksuriuri started May 3, 2024 in Show and tell

  

  

  2
• 1 You must be logged in to vote

  🙏

  Difficulty understanding sequence length and context length

  NicolasDrapier asked May 6, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  Offline Batched Inference response issue

  NihinduCG started May 6, 2024 in General

  

  

  0
• 2 You must be logged in to vote

  🙏

  Printing from CUDA code

  gpu2024 asked May 6, 2024 in Q&A · Unanswered

  

  

  0
• 5 You must be logged in to vote

  🙏

  Is there any way that vllm works with huggingface pipeline?

  linkfqy asked Mar 31, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  RuntimeError with CUDA 12.2 on Windows using vLLM and Llava: No Kernel Image Available

  OualidBougzime started Apr 26, 2024 in General · Closed

  

  

  1
• 1 You must be logged in to vote

  🙏

  Has anyone tried to use some modules from VLLM to replace torch.nn.functional.scaled_dot_product_attention?

  leprodeveloper asked May 2, 2024 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  Getting an assertion error on llama3 70b

  tsvisab asked May 1, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Issue Running LLaVA with vLLM Due to Tensor Size Mismatch

  OualidBougzime asked Apr 28, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  how to use auth token to use llama2 in vLLM? and question about presence_penalty

  orellavie1212 asked Aug 9, 2023 in Q&A · Closed · Unanswered

  

  

  3