[Bug] xunfei api 评测报错

[wtlwang2019]

先决条件

• 我已经搜索过 问题 和 讨论 但未得到预期的帮助。
• 错误在 最新版本 中尚未被修复。

问题类型

我正在使用官方支持的任务/模型/数据集进行评估。

环境

Ubuntu 22.04
miniconda3
opencompass[api]

重现问题 - 代码/配置示例

import os
from opencompass.models import XunFeiSpark, XunFei
from opencompass.partitioners import NaivePartitioner
from opencompass.runners.local_api import LocalAPIRunner
from opencompass.tasks import OpenICLInferTask

你前面获得的 api 服务地址

model_url = 'wss://spark-api.xf-yun.com/v1.1/chat' # lite
api_key = os.getenv('X_API_KEY')
api_meta_template = dict(
round=[
dict(role='HUMAN', api_role='HUMAN'),
dict(role='BOT', api_role='BOT', generate=True),
],
reserved_roles=[dict(role='SYSTEM', api_role='SYSTEM'),]
)

models = [ dict(
type=XunFeiSpark,
path='lite', # 请求服务时的 model name
url=model_url, # 服务地址
app_id='xxx',
api_key='xxx',
api_secret=api_key, # API key
query_per_second=0.3, # 服务请求速率
max_seq_len=4096, # 最大输入长度
# max_out_len=1024, # 最大输出长度
# meta_template=api_meta_template,
# rpm_verbose=True, # 是否打印请求速率
# temperature=0.1, # 生成温度
# batch_size=1, # 批处理大小
retry=2, # 重试次数
# extra_body={"stream": False},
# logprobs = True,
# verbose=True
)]

infer = dict(
partitioner=dict(type=NaivePartitioner),
runner=dict(
type=LocalAPIRunner,
max_num_workers=2,
concurrent_users=1,
task=dict(type=OpenICLInferTask)),
)

重现问题 - 命令或脚本

opencompass --models xunfei_api --datasets demo_gsm8k_chat_gen --debug

重现问题 - 错误信息

(opencompass) wtl@iZf8z1mesjci0f2gxm0hkbZ:~/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs$ opencompass --models xunfei_api --datasets demo_math_base_gen --debug
03/05 09:52:40 - OpenCompass - INFO - Loading demo_math_base_gen: /home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs/./datasets/demo/demo_math_base_gen.py
03/05 09:52:40 - OpenCompass - INFO - Loading xunfei_api: /home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs/./models/xunfei/xunfei_api.py
03/05 09:52:40 - OpenCompass - INFO - Loading example: /home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/configs/./summarizers/example.py
03/05 09:52:40 - OpenCompass - INFO - Current exp folder: outputs/default/20250305_095240
03/05 09:52:40 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
03/05 09:52:40 - OpenCompass - INFO - Partitioned into 1 tasks.
03/05 09:52:41 - OpenCompass - INFO - Task [X-lite/demo_math]
03/05 09:52:41 - OpenCompass - INFO - Try to load the data from /home/wtl/.cache/opencompass/./data/math/
03/05 09:52:41 - OpenCompass - INFO - Start inferencing [X-lite/demo_math]
[2025-03-05 09:52:42,017] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting build dataloader
[2025-03-05 09:52:42,017] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
0%| | 0/64 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/wtl/miniconda3/envs/opencompass/bin/opencompass", line 8, in
sys.exit(main())
File "/home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/cli/main.py", line 308, in main
runner(tasks)
File "/home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/runners/base.py", line 38, in call
status = self.launch(tasks)
File "/home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/runners/local.py", line 128, in launch
task.run(cur_model=getattr(self, 'cur_model',
File "/home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/tasks/openicl_infer.py", line 89, in run
self._inference()
File "/home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/tasks/openicl_infer.py", line 134, in _inference
inferencer.inference(retriever,
File "/home/wtl/miniconda3/envs/opencompass/lib/python3.10/site-packages/opencompass/openicl/icl_inferencer/icl_gen_inferencer.py", line 140, in inference
entry, golds = list(zip(*datum))
ValueError: not enough values to unpack (expected 2, got 1)

其他信息

使用xunfei api 模式评测报错

[MaiziXiao]

We will have a look into the problem. For the moment, please try to use OpenAISDK for the Xunfei model evaluation, it will be a more general solution for API model evaluation.