[Bug] AttributeError: 'ConfigDict' object has no attribute 'reader_cfg'

[xtuhcy]

先决条件

• 我已经搜索过 问题 和 讨论 但未得到预期的帮助。
• 错误在 最新版本 中尚未被修复。

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

{'CUDA available': False,
'GCC': 'n/a',
'MMEngine': '0.10.5',
'MSVC': '用于 x64 的 Microsoft (R) C/C++ 优化编译器 19.29.30141 版',
'MUSA available': False,
'OpenCV': '4.10.0',
'PyTorch': '2.4.1+cpu',
'PyTorch compiling details': 'PyTorch built with:\n'
' - C++ Version: 201703\n'
' - MSVC 192930154\n'
' - Intel(R) oneAPI Math Kernel Library Version '
'2024.2.1-Product Build 20240722 for Intel(R) 64 '
'architecture applications\n'
' - Intel(R) MKL-DNN v3.4.2 (Git Hash '
'1137e04ec0b5251ca2b4400a4fd3c667ce843d67)\n'
' - OpenMP 2019\n'
' - LAPACK is enabled (usually provided by '
'MKL)\n'
' - CPU capability usage: AVX2\n'
' - Build settings: BLAS_INFO=mkl, '
'BUILD_TYPE=Release, '
'CXX_COMPILER=C:/actions-runner/_work/pytorch/pytorch/builder/windows/tmp_bin/sccache-cl.exe, '
'CXX_FLAGS=/DWIN32 /D_WINDOWS /GR /EHsc '
'/Zc:__cplusplus /bigobj /FS /utf-8 '
'-DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO '
'-DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER '
'-DUSE_FBGEMM -DUSE_XNNPACK '
'-DSYMBOLICATE_MOBILE_DEBUG_HANDLE /wd4624 '
'/wd4068 /wd4067 /wd4267 /wd4661 /wd4717 /wd4244 '
'/wd4804 /wd4273, LAPACK_INFO=mkl, '
'PERF_WITH_AVX=1, PERF_WITH_AVX2=1, '
'PERF_WITH_AVX512=1, TORCH_VERSION=2.4.1, '
'USE_CUDA=0, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, '
'USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, '
'USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, '
'USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, '
'USE_NNPACK=OFF, USE_OPENMP=ON, USE_ROCM=OFF, '
'USE_ROCM_KERNEL_ASSERT=OFF, \n',
'Python': '3.10.15 | packaged by Anaconda, Inc. | (main, Oct 3 2024, '
'07:22:19) [MSC v.1929 64 bit (AMD64)]',
'TorchVision': '0.19.1+cpu',
'lmdeploy': "not installed:No module named 'lmdeploy'",
'numpy_random_seed': 2147483648,
'opencompass': '0.3.3+22a4e76',
'sys.platform': 'win32',
'transformers': '4.45.2'}

重现问题 - 代码/配置示例

from mmengine.config import read_base

with read_base():
from .datasets.asp.demo_dataset import asp_demo_datasets
from .models.openai.asp_info_api import models as asp_info_api_models

datasets = asp_demo_datasets
models = asp_info_api_models

from opencompass.datasets.custom import CustomDataset

asp_demo_datasets = [
dict(
type=CustomDataset,
abbr='asp_demo',
path='demo_eval_data.jsonl'
)
]

from opencompass.models import OpenAISDK

api_meta_template = dict(round=[
dict(role='HUMAN', api_role='HUMAN'),
dict(role='BOT', api_role='BOT', generate=True),
], )

models = [
dict(abbr='asp_info_api',
type=OpenAISDK,
openai_api_base="http://xxx:xxx/v1/chat/completions",
path='72B-Chat',
key='st-*',
meta_template=api_meta_template,
query_per_second=1,
batch_size=1,
temperature=0.01,
max_completion_tokens=8192,
verbose=True),
]

重现问题 - 命令或脚本

python run.py configs/eval_asp_audit.py --debug

重现问题 - 错误信息

signal.SIGALRM is not available on this platform
signal.SIGALRM is not available on this platform
10/09 15:15:13 - OpenCompass - INFO - Current exp folder: outputs\default\20241009_151513
10/09 15:15:13 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
Traceback (most recent call last):
File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\mmengine\config\config.py", line 109, in getattr
value = super().getattr(name)
File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\addict\addict.py", line 67, in getattr
return self.getitem(item)
File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\mmengine\config\config.py", line 138, in getitem
return self.build_lazy(super().getitem(key))
File "D:\miniconda3\envs\audit_llm_eval\lib\site-packages\mmengine\config\config.py", line 105, in missing
raise KeyError(name)
KeyError: 'reader_cfg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\python_src\eval_opencompass\run.py", line 4, in
main()
File "D:\python_src\eval_opencompass\opencompass\cli\main.py", line 299, in main
tasks = partitioner(cfg)
File "D:\python_src\eval_opencompass\opencompass\partitioners\base.py", line 83, in call
tasks = self.partition(**model_and_dataset_args,
File "D:\python_src\eval_opencompass\opencompass\partitioners\num_worker.py", line 69, in partition
dataset_size = self.get_size(dataset)
File "D:\python_src\eval_opencompass\opencompass\partitioners\num_worker.py", line 134, in get_size
test_range = dataset.reader_cfg.get('test_range', '')
File "D:\miniconda3\envs\llm_eval\lib\site-packages\mmengine\config\config.py", line 113, in getattr
raise AttributeError(f"'{self.class.name}' object has no "
AttributeError: 'ConfigDict' object has no attribute 'reader_cfg'

其他信息

No response

[tonysy]

Please check the confis/datasets for more example, reader_cfg are required for dataset loading.