求助: 4060TI 16G 部署无论怎么调整max_split_size_mb都会在加载第四个checkpoint后报 CUDA out of memory.

[luguoj]

求助: 4060TI 16G 部署无论怎么调整max_split_size_mb都会在加载第四个checkpoint后报 CUDA out of memory.
7个checkpoint都完整性校验过都是一致的

[luguoj]

设置max_split_size_mb为214以下时，加载第三个checkpoint后报 ...Tried to allocate 214.00 MiB....
设置为215时，加载第四个checkpoint后，报错如下
报错：torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 108.00 MiB. GPU 0 has a total capacity of 16.00 GiB of which 7.62 GiB is free. Of the allocated memory 7.25 GiB is allocated by PyTorch, and 1.56 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

[luguoj]

解决了。。。使用的是cli_demo.py
把

model = AutoModel.from_pretrained(MODEL_PATH, trust_remote_code=True, device_map="auto").eval()

这行的device_map="auto"删掉，不需要设置max_split_size_mb直接过了