Errors when run zsre on GPT2xl

[ZihaoLin0123]

Thanks for your great work!

I am trying to run zsre (qa) task using GPT2xl model using the default config, but get some errors.

Error executing job with overrides: ['+alg=lora', '+experiment=qa', '+model=gpt2xl']
Traceback (most recent call last):
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/zihao/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/main.py", line 39, in
cli.main()
File "/home/zihao/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
run()
File "/home/zihao/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
runpy.run_path(target, run_name="main")
File "/home/zihao/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
return _run_module_code(code, init_globals, run_name,
File "/home/zihao/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/home/zihao/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
exec(code, run_globals)
File "/home/zihao/memory-editing/MELO/melo/run.py", line 135, in
run()
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/main.py", line 48, in decorated_main
_run_hydra(
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
run_and_report(
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
raise ex
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
return func()
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/_internal/utils.py", line 378, in
lambda: hydra.run(
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 111, in run
_ = ret.return_value
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/core/utils.py", line 233, in return_value
raise self._return_value
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/hydra/core/utils.py", line 160, in run_job
ret.return_value = task_function(task_cfg)
File "/home/zihao/memory-editing/MELO/melo/run.py", line 131, in run
trainer.run_edit()
File "/home/zihao/memory-editing/MELO/melo/trainer.py", line 174, in run_edit
self.alg.edit(tokens)
File "/home/zihao/memory-editing/MELO/melo/algs/lora.py", line 146, in edit
outputs = self.model.model(**tokens)
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1109, in forward
loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/torch/nn/modules/loss.py", line 1174, in forward
return F.cross_entropy(input, target, weight=self.weight,
File "/home/zihao/anaconda3/envs/EasyEdit/lib/python3.10/site-packages/torch/nn/functional.py", line 3029, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
ValueError: Expected input batch_size (1900) to match target batch_size (500).

Have you tried to run zsre on GPT2xl models? Or do you have some suggestions on this?

Thanks!

[BruthYU]

Thanks for your interest 😸
Currently MELO only supports T5 on the zsRE task. However, it is possible to do this via modifying the code.

Necessary modifications I can think of for now:

• melo/config/experiment/qa.yaml (e.g. the lora_task_type and other block configurations)
• Track the forward pass in peft_egg/src/tuners/melo.py and debug the errors.

We'll try GPT2-XL on zsRE in the future, please stay tuned to our later releases ⌛ 👨‍💻