Build vllm without recompiling pytorch?

[prarit]

I am building a series of tools and have already built torch-2.3.0. Is there a way to have vllm use that pytorch instead of recompiling torch-2.3.0?