Vicuna-13B results

[yzc111]

Hello, when I reproduce the results on Vicuna-13B and Llams2-7B , I can not get any model output, and the code outputs the warning:"Prompt exceeds max length and return an empty string as answer. If this happens too many times, it is suggested to make the prompt shorter", How to deal with this phenomenon? Thank you~

[gaotianyu1350]

Hi,

Which config are you using? Vicuna and llama2 models have a 4k context window limit, which limits how many passages you can use in the context.

[yzc111]

Hi, thank you for your reply, the config is 2 shot, 3 ndoc

[gaotianyu1350]

Did you use the "light instruction" version as well?

[yzc111]

NO, I just use the default setting

[gaotianyu1350]

Can you try this config (but change the model name): https://github.com/princeton-nlp/ALCE/blob/main/configs/asqa_alpaca-7b_shot2_ndoc3_gtr_light_inst.yaml

[yzc111]

OK. thanks~

[yzc111]

another question， when I use the setting
prompt_file: prompts/asqa_light_inst.json
eval_file: data/asqa_eval_gtr_top100.json
shot: 2
ndoc: 3
dataset_name: asqa
tag: gtr_light_inst
model: vicuna-13b
temperature: 1.0
top_p: 0.95
to reproduce the result, I get the QA-EM=19.7 and mauve=70.7. the paper reports EM=31.9 mauve=82.6. are there any different settings in the config file?

[howard-yen]

Note that there is a difference between EM and QA-EM, and we report EM in the paper. Can you post the full output or .score file? Can you also post the link to the vicuna model that you are using? There are a couple different versions with different performances.

[yzc111]

Hi. this is the config of we used to reproduce the result on vicuna-13B
prompt_file: prompts/asqa_light_inst.json
eval_file: data/asqa_eval_gtr_top100.json
shot: 2
ndoc: 3
dataset_name: asqa
tag: gtr_light_inst
model: /work/models/vicuna-13b
temperature: 1.0
top_p: 0.95

[yzc111]

so, how can I get the EM score of your paper reported?

[gaotianyu1350]

That is “str_em"

[yzc111]

Fine，Thanks