encode_special_tokens=True 导致防注入失效 #415
-
|
为了适配fschat，tokenization_chatglm.py加了encode_special_tokens=True后，防注入就失效了，多轮对话可以拼接成一个句子，然后在转化为id，<|user|>就是一个token。而防注入希望，当用户输入<|user|>，会被tokenize 成多个token。 |
Beta Was this translation helpful? Give feedback.
Answered by
zRzRzRzRzRzRzR
Nov 29, 2023
Replies: 1 comment
Answer selected by
zRzRzRzRzRzRzR
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
mark