➕ Install peft for prompt adapter support #124
Conversation
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: prashantgupta24 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
|
New changes are detected. LGTM label has been removed. |
Signed-off-by: Prashant Gupta <[email protected]>
Signed-off-by: Prashant Gupta <[email protected]>
| @@ -180,6 +180,10 @@ RUN --mount=type=cache,target=/root/.cache/pip \ | |||
| --mount=type=cache,target=/root/.cache/uv \ | |||
| uv pip install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.1.2/flashinfer-0.1.2+cu121torch2.4-cp311-cp311-linux_x86_64.whl | |||
|
|
|||
| # Install peft for prompt adapter support | |||
There was a problem hiding this comment.
Sorry if I get back to this, but I just realized this is something which is only supported in the adapter, right? In that case this should be a dependency in vllm-tgis-adapter and not something installed here.
There was a problem hiding this comment.
This is something that is needed by vllm explicitly -
vllm/vllm/prompt_adapter/models.py
Line 93 in 7a21f52
There was a problem hiding this comment.
Also see these lines within PromptAdapterConfig:
Lines 1334 to 1340 in 7a21f52
There was a problem hiding this comment.
I can confirm that this is needed to run the upstream unit tests.
|
Will try to get it merged in upstream https://github.com/vllm-project/vllm/pull/8123/files |
* add memory clean up after every shape and parameter to reduce cache invalidation buffers * small typo * syntax change --------- Co-authored-by: maleksan85 <[email protected]>
For allowing PEFT adapters to be loaded, we will need the
peftlibrary installed.Upstream PR: vllm-project#8123