Loading models from an S3 location instead of local path

[petrosbaltzis]

Hello,

The VLLM library gives the ability to load the model and the tokenizer either from a local folder or directly from HuggingFace.

["python", "-m", "vllm.entrypoints.openai.api_server", \
"--host=0.0.0.0", \
"--port=8080", \
"--model=<local_path>", \
"--tokenizer=<local_path>",
]

I wonder if this functionality can be extended to support s3 locations so that when we initialize the API server, we pass the proper S3 location.

["python", "-m", "vllm.entrypoints.openai.api_server", \
"--host=0.0.0.0", \
"--port=8080", \
"--model=<s3://bucket/prefix>", \
"--tokenizer=<s3://bucket/prefix>",
]

Petros

[ikalista]

My thin opinion: you may wrap the s3 storage with something like juiceFS, or use some kind of mounting technique, making s3 indistinguishable from a regular path. Instead of letting vllm do this compatibility. We use NAS or juiceFS wrapped s3 storage/ceph works fine with vllm😀

[simon-mo]

Good point. Moved this to an issue #3090