Contiguous PA

[mfylcek]

Contiguous cache fetching to avoid using costly gather operation. Requires changes in vllm-hpu-extension (HabanaAI/vllm-hpu-extension#17) to work.

Introduces redundant calculations in decoding phase. In all tested cases improves performance over the entire run (5-12%). For even better performance cache defragmentation is required. Only compatible with v2-block-manager.

[xuechendi]

Hi, @mfylcek , we've been follow this branch, just did test on Gaudi3 - static batch_size as 128

test script:

python -m vllm.entrypoints.openai.api_server --port 8080 --model meta-llama/Meta-Llama-3.1-8B-Instruct --tensor-parallel-size 1 --max-num-seqs 128 --disable-log-requests --dtype bfloat16 --block-size 128 --gpu-memory-util 0.9 --num-lookahead-slots 1 --use-v2-block-manager  --max-model-len 4096

# repeat command below 3 times for warming up then get final result
python benchmark_serving.py --backend vllm --model meta-llama/Meta-Llama-3.1-8B-Instruct --dataset-name sonnet --dataset-path ./sonnet.txt --request-rate 512 --num-prompts 56 --port 8080 --sonnet-input-len 1024 --sonnet-output-len 1024 --sonnet-prefix-len 100

[xuechendi]

@mfylcek @michalkuligowski ,
our team submitted this PR: #426 which will effectively reduce fragmentation when creating block_list.

Here is the performance I measured with PR 426
data deleted

[xuechendi]

From observation, after warm up,

• BlockList with only PR424 will be double-sized with lots of [-1]
• Blocklist with PR424 + PR426, blocklist size will be as same as previous run