Why do not you calculate the real output token for throughput?

[wqh17101]

vllm/benchmarks/benchmark_throughput.py

Lines 177 to 180 in c894836

	total_num_tokens = sum(
	prompt_len + output_len
	for _, prompt_len, output_len in requests
	)

You can see that you just use the tokens' length of request in the dataset , not the real output length for the model to eval.

[Godricly]

The token returned also contains input prompt. I don't think computing the output will solve you problem.

vllm/vllm/entrypoints/api_server.py

Lines 64 to 67 in c6dfc3c

	prompt = final_output.prompt
	text_outputs = [prompt + output.text for output in final_output.outputs]
	ret = {"text": text_outputs}
	return JSONResponse(ret)

[zhuohan123]

I believe the main reason is that prompt tokens also take compute. There are different ways to measure throughput for LLMs, but I believe the trends between systems should be similar under different metrics. Move this issue to discussions for future questions.

[zhaoyang-star]

Yes. It is an open discussion whether or not take prompt tokens into account. As far as I know, the latency metric is not included the length of prompt tokens.