You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm evaluating with the officially supported tasks/models/datasets.
Environment
eval_hf_qwen2_5_0_5b_instruct.py文件内容如下
from mmengine.config import read_base
with read_base():
from opencompass.configs.datasets.gsm8k.gsm8k_gen_3309bd import gsm8k_datasets
from opencompass.configs.models.qwen2_5.hf_qwen2_5_0_5b_instruct import models
from opencompass.configs.summarizers.example import summarizer
datasets = sum([v for k, v in locals().items() if k.endswith('_datasets') or k == 'datasets'], [])
work_dir = './outputs/hf_qwen2_5_0_5b_instruct/'
eval_vllm_qwen2_5_0_5b_instruct.py文件内容如下
from mmengine.config import read_base
with read_base():
from opencompass.configs.datasets.gsm8k.gsm8k_gen_3309bd import gsm8k_datasets
from opencompass.configs.models.qwen2_5.vllm_qwen2_5_0_5b_instruct import models
from opencompass.configs.summarizers.example import summarizer
datasets = sum([v for k, v in locals().items() if k.endswith('_datasets') or k == 'datasets'], [])
work_dir = './outputs/hf_qwen2_5_0_5b_instruct/'
Prerequisite
Type
I'm evaluating with the officially supported tasks/models/datasets.
Environment
eval_hf_qwen2_5_0_5b_instruct.py文件内容如下
from mmengine.config import read_base
with read_base():
from opencompass.configs.datasets.gsm8k.gsm8k_gen_3309bd import gsm8k_datasets
from opencompass.configs.models.qwen2_5.hf_qwen2_5_0_5b_instruct import models
from opencompass.configs.summarizers.example import summarizer
datasets = sum([v for k, v in locals().items() if k.endswith('_datasets') or k == 'datasets'], [])
work_dir = './outputs/hf_qwen2_5_0_5b_instruct/'
eval_vllm_qwen2_5_0_5b_instruct.py文件内容如下
from mmengine.config import read_base
with read_base():
from opencompass.configs.datasets.gsm8k.gsm8k_gen_3309bd import gsm8k_datasets
from opencompass.configs.models.qwen2_5.vllm_qwen2_5_0_5b_instruct import models
from opencompass.configs.summarizers.example import summarizer
datasets = sum([v for k, v in locals().items() if k.endswith('_datasets') or k == 'datasets'], [])
work_dir = './outputs/hf_qwen2_5_0_5b_instruct/'
hf评测命令
CUDA_VISIBLE_DEVICES=6 python3 run.py configs/eval_hf_qwen2_5_0_5b_instruct.py --debug
vllm 评测命令
CUDA_VISIBLE_DEVICES=6 python3 run.py configs/eval_vllm_qwen2_5_0_5b_instruct.py --debug
Reproduces the problem - code/configuration sample
hf评测命令
CUDA_VISIBLE_DEVICES=6 python3 run.py configs/eval_hf_qwen2_5_0_5b_instruct.py --debug
vllm 评测命令
CUDA_VISIBLE_DEVICES=6 python3 run.py configs/eval_vllm_qwen2_5_0_5b_instruct.py --debug
Reproduces the problem - command or script
hf评测命令
CUDA_VISIBLE_DEVICES=6 python3 run.py configs/eval_hf_qwen2_5_0_5b_instruct.py --debug
vllm 评测命令
CUDA_VISIBLE_DEVICES=6 python3 run.py configs/eval_vllm_qwen2_5_0_5b_instruct.py --debug
Reproduces the problem - error message
hf结果:
dataset,version,metric,mode,qwen2.5-0.5b-instruct-hf
gsm8k,3309bd,accuracy,gen,0.83
vllm 结果:
dataset,version,metric,mode,qwen2.5-0.5b-instruct-vllm
gsm8k,3309bd,accuracy,gen,1.44
Other information
No response
The text was updated successfully, but these errors were encountered: