Vllm's sampling significantly more deterministic than HF?

[syskn]

Hi guys, great work!

I have been experimenting with the library for several weeks, and immediately noticed that sampled tokens (with the same temperature and such) are significantly more deterministic with Vllm vs. HF Transformers using the same models - with temperature lower than 0.7, often the first 5-10 sampled tokens are exactly same across few different generations, even recreating the original text in the datasets verbatim, like there is some greedy decoding going on (when it is not). This unfortunately leads to a significant repetition issue I've never seen with HF.

Initially I thought this has to do with special tokens (such as </s>) but it was not the case.

In the mean time, I have also been checking and modifying codebase to see if there is any discrepancy between the sampling process, but I am not sure about the difference.

Does anyone else experienced a similar behavior?

Might be related: #450

[81549361]

Yes, I have also encountered the same prompt, the same sentence will be generated

[BaiMoHan]

Yes, I've encountered the same problem too. I would also like to know how to solve it.

[anujnayyar1]

Same here.
Modifying the presence and frequency penalties appeared to help.
However they cause the model output to degridate significantly.

[syskn]

#590
Related topic with actual examples. The topic is about GPTJ, but the issue is there with other architectures such as Llama and NeoX.

To be clear, the repetitive/potential degradation issue is not related to special tokens such as <s></s>. I modified vllm so that it never generates those special tokens like HF's bad_words_ids, but the issue persists (those special tokens will also make the inference quality significantly worse especially with non-chat prompts, but it is a different issue).

[ehartford]

I have seen this behavior myself. It's blocking me.
Is there a reason this is a discussion instead of an issue?
It's bad enough to force me to use TGI instead

[syskn]

Good point. I created a new issue.

[ri938]

I also noticed this issue. I think it can be solved by someone implementing the option for a HF sampler. It should be possible to match exact performance.