Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix one_hot bug in torch compile mode #427

Merged
merged 4 commits into from
Oct 29, 2024
Merged

Fix one_hot bug in torch compile mode #427

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7fc910f
vllm/worker/hpu_model_runner.py
yuwenzho Oct 25, 2024
44583b5
Update vllm/worker/hpu_model_runner.py
yuwenzho Oct 28, 2024
91ab89b
update vllm/worker/hpu_model_runner.py
yuwenzho Oct 28, 2024
bce42f3
remove redundant code
yuwenzho Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions vllm/worker/hpu_model_runner.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -317,18 +317,19 @@ def _set_block_mapping(self, metadata, batch_size, device, dtype):
mask = mask >= metadata.block_usage.unsqueeze(-1)
attn_bias = (torch.zeros_like(mask, dtype=dtype).masked_fill_(
mask, -math.inf))
if is_fake_hpu():
# Unfortunately one_hot on CPU doesn't handle
# out of bounds classes. We need to mask those
# values manually
oob_values = metadata.block_mapping.lt(0)
block_mapping = metadata.block_mapping.masked_fill(oob_values, 0)
block_mapping = torch.nn.functional.one_hot(block_mapping,

if not is_fake_hpu() and htorch.utils.internal.is_lazy():
block_mapping = torch.nn.functional.one_hot(metadata.block_mapping,
num_classes=batch_size)
block_mapping.masked_fill_(oob_values.unsqueeze(-1), 0)
else:
block_mapping = torch.nn.functional.one_hot(metadata.block_mapping,
# Unfortunately one_hot on CPU/torch.compile mode/eager mode
# doesn't handle out of bounds classes,
# so we convert all negative values to 0.
block_mapping = torch.nn.functional.relu(metadata.block_mapping)
block_mapping = torch.nn.functional.one_hot(block_mapping,
num_classes=batch_size)
oob_values = metadata.block_mapping.lt(0)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this and next line were left accidentally ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and next line cannot be deleted. Negative values in metadata.block_mapping must be represented as zeros after one_hot, otherwise it will cause error in further processing.
Here is an example for better understanding:
metadata.block_mapping=[0, -1, -1] --relu()--> [0, 0, 0] --one_hot with num_classes=1--> [[1], [1], [1]] --oob value mask--> [[1], [0], [0]]

block_mapping.masked_fill_(oob_values.unsqueeze(-1), 0)
block_mapping = block_mapping.to(dtype)
metadata = metadata._replace(block_mapping=block_mapping,
attn_bias=attn_bias)
Expand Down
Loading