-
Notifications
You must be signed in to change notification settings - Fork 406
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] AWQ量化InternVL2 20B输出无意义的杂乱文本 #2650
Comments
理解的vl模型的量化和lm模型的AWQ量化有差异吧,AWQ量化需要使用数据集进行处理,但是vl的输入是images feature + query等信息,包括embedding,直接使用query进行量化,效果应该会很差吧 |
回答的压根不是句子,我看教程只用lmdeploy lite auto_awq 量化的,不知道哪一步错了 |
1 similar comment
回答的压根不是句子,我看教程只用lmdeploy lite auto_awq 量化的,不知道哪一步错了 |
没看懂,issue 里面说是 internvl 模型,但是命令给的都是 internlm。可以排查下变量:
|
抱歉,路径写错了,已更正,正准备用原始模型试试;步骤是正确的吗(只需执行lmdeploy lite auto_awq,无需其他操作) |
量化时出现这句有没有影响 |
是不是你训的模型 tokenizer 有点问题?1085165> 4096 差距有点大了,其他模型也有,但是量化也没啥问题 |
Checklist
Describe the bug
训练后的模型,使用AWQ量化InternVL2-20B输出无意义的杂乱文本
Reproduction
量化(因为开发环境没有互联网,pth_text_only通过脚本下下载的)
下载脚本
from datasets import load_dataset
traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')
lmdeploy lite auto_awq
/root/models/internvl2-26B
--calib-dataset 'ptb'
--calib-samples 128
--calib-seqlen 2048
--w-bits 4
--w-group-size 128
--work-dir /root/models/internvl2-26B_awq_4bit
运行
lmdeploy serve api_server
/root/models/internvl2-26B_awq_4bit
--server-name 0.0.0.0
--server-port 23333
--tp2
Environment
Error traceback
The text was updated successfully, but these errors were encountered: