Add sample chat template into vLLM container #152
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vaibhavjainwiz The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
@vaibhavjainwiz: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
vaibhavjainwiz
force-pushed
the
RHOAIENG-12233
branch
from
45a275f
to
9422ba0
Compare
|
/lgtm |
|
/cherry-pick release |
|
@vaibhavjainwiz: only opendatahub-io org members may request cherry picks. If you are already part of the org, make sure to change your membership to public. Otherwise you can still do the cherry-pick manually. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/cherry-pick release |
|
@vaibhavjainwiz: new pull request created: #153 In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
* add nccl performance environment variable for mi300x * remove export * add it to the perf doc
FILL IN THE PR DESCRIPTION HERE
Jira issue : https://issues.redhat.com/browse/RHOAIENG-12233
The built-in Transformers v4.44 is shipped with vLLM v0.5.5. Transformers v4.44 has a strict requirement that a chat template must be provided with the model. While many chat models already follow this practice, some models are not working due to the absence of a chat template.
For models without an in-built chat template, vLLM exposes a command-line parameter (chat_template), which can be passed along with other parameters in ServingRuntime. In addition to this, the vLLM community has provided several sample templates in their examples directory, which users can select based on their model type.
From the user’s perspective, if their model lacks an in-built template, they must follow these steps based on their model type:
Find the appropriate sample template from the vLLM examples directory.
Mount the selected template on the vLLM pod.
Provide the template path using the vLLM command-line argument.
To simplify this process, we are streamlining steps 1 and 2. We will add the sample templates provided in the vLLM examples directory directly into the ODH vLLM container images. This ensures that all templates are pre-existing within the vLLM container, and users only need to configure the vLLM command-line argument to point to the correct template path.
FIX #xxxx (link existing issues this PR will resolve)
BEFORE SUBMITTING, PLEASE READ THE CHECKLIST BELOW AND FILL IN THE DESCRIPTION ABOVE
PR Checklist (Click to Expand)
Thank you for your contribution to vLLM! Before submitting the pull request, please ensure the PR meets the following criteria. This helps vLLM maintain the code quality and improve the efficiency of the review process.
PR Title and Classification
Only specific types of PRs will be reviewed. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:
[Bugfix]for bug fixes.[CI/Build]for build or continuous integration improvements.[Doc]for documentation fixes and improvements.[Model]for adding a new model or improving an existing model. Model name should appear in the title.[Frontend]For changes on the vLLM frontend (e.g., OpenAI API server,LLMclass, etc.)[Kernel]for changes affecting CUDA kernels or other compute kernels.[Core]for changes in the core vLLM logic (e.g.,LLMEngine,AsyncLLMEngine,Scheduler, etc.)[Hardware][Vendor]for hardware-specific changes. Vendor name should appear in the prefix (e.g.,[Hardware][AMD]).[Misc]for PRs that do not fit the above categories. Please use this sparingly.Note: If the PR spans more than one category, please include all relevant prefixes.
Code Quality
The PR need to meet the following code quality standards:
format.shto format your code.docs/source/if the PR modifies the user-facing behaviors of vLLM. It helps vLLM user understand and utilize the new features or changes.Notes for Large Changes
Please keep the changes as concise as possible. For major architectural changes (>500 LOC excluding kernel/data/config/test), we would expect a GitHub issue (RFC) discussing the technical design and justification. Otherwise, we will tag it with
rfc-requiredand might not go through the PR.What to Expect for the Reviews
The goal of the vLLM team is to be a transparent reviewing machine. We would like to make the review process transparent and efficient and make sure no contributor feel confused or frustrated. However, the vLLM team is small, so we need to prioritize some PRs over others. Here is what you can expect from the review process:
action-requiredlabel on the PR if there are changes required. The contributor should address the comments and ping the reviewer to re-review the PR.Thank You
Finally, thank you for taking the time to read these guidelines and for your interest in contributing to vLLM. Your contributions make vLLM a great tool for everyone!