ValueError: You are trying to offload the whole model to the disk. Please use the disk_offload function instead.`

[1105135335]

(ht240815) PS G:\project\ht\240815\LongWriter> python .\trans_web_demo.py Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<?, ?it/s] Traceback (most recent call last): File "G:\project\ht\240815\LongWriter\trans_web_demo.py", line 59, in <module> model, tokenizer = load_model_and_tokenizer(MODEL_PATH, trust_remote_code=True) File "G:\project\ht\240815\LongWriter\trans_web_demo.py", line 49, in load_model_and_tokenizer model = AutoModelForCausalLM.from_pretrained( File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained return model_class.from_pretrained( File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\modeling_utils.py", line 4015, in from_pretrained dispatch_model(model, **device_map_kwargs) File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\accelerate\big_modeling.py", line 496, in dispatch_model raise ValueError( ValueError: You are trying to offload the whole model to the disk. Please use the disk_offload function instead.

昨晚还一切正常，运行起来了，同样的代码，啥也没动，今个一直报这个。下面是transformers的版本：

> pip show transformers Name: transformers Version: 4.44.0 Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow Home-page: https://github.com/huggingface/transformers Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors) Author-email: transformers@huggingface.co License: Apache 2.0 License Location: f:\tool\anaconda3\envs\ht240815\lib\site-packages Requires: filelock, huggingface-hub, numpy, packaging, pyyaml, regex, requests, safetensors, tokenizers, tqdm Required-by: peft

[1105135335]

初学者，还请见谅

[bys0318]

Hi, can you try the solution here: huggingface/accelerate#2726?

[1105135335]

试了试，解决了。非常感谢。

所以，这个没问题，我可以理解为是因为没有GPU的原因吗？

[1105135335]

第一次出现这个问题，修改的部分如下：

        # model = AutoModelForCausalLM.from_pretrained(
        #     model_dir, trust_remote_code=trust_remote_code, device_map='auto'
        # )
        model = transformers.pipeline(
            "text-generation",
            model=model_dir,
            model_kwargs={"torch_dtype": torch.bfloat16},
            device_map="auto",
        )

结束程序并未做任何修改，又再一次重新在conda虚拟环境的控制台输入python trans_web_demo.py运行程序，和上次运行同样的步骤，同样的环境，同样的代码，又报了如下这个错误：

Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "G:\project\ht\240815\LongWriter\trans_web_demo.py", line 68, in <module>
    model, tokenizer = load_model_and_tokenizer(MODEL_PATH, trust_remote_code=True)
  File "G:\project\ht\240815\LongWriter\trans_web_demo.py", line 55, in load_model_and_tokenizer
    model = transformers.pipeline(
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\pipelines\__init__.py", line 895, in pipeline
    framework, model = infer_framework_load_model(
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\pipelines\base.py", line 299, in infer_framework_load_model
    raise ValueError(
ValueError: Could not load model G:\project\ht\240815\LongWriter\THUDM\LongWriter-llama3.1-8b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.llama.modeling_llama.LlamaForCausalLM'>). See the original errors:

while loading with AutoModelForCausalLM, an error is thrown:
Traceback (most recent call last):
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\pipelines\base.py", line 286, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\models\auto\auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\modeling_utils.py", line 4015, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\accelerate\big_modeling.py", line 496, in dispatch_model
    raise ValueError(
ValueError: You are trying to offload the whole model to the disk. Please use the `disk_offload` function instead.

while loading with LlamaForCausalLM, an error is thrown:
Traceback (most recent call last):
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\pipelines\base.py", line 286, in infer_framework_load_model
    model = model_class.from_pretrained(model, **kwargs)
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\transformers\modeling_utils.py", line 4015, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "F:\Tool\Anaconda3\envs\ht240815\lib\site-packages\accelerate\big_modeling.py", line 496, in dispatch_model
    raise ValueError(
ValueError: You are trying to offload the whole model to the disk. Please use the `disk_offload` function instead.

[bys0318]

你好，我们更新了模型的modeling_chatglm.py以及github中的trans_web_demo.py，请试一下更新之后是否还有问题？

[1105135335]

非常真诚的表达感谢，我是新手，还望见谅，提的问题可能会比较简单且奇怪，实在是抱歉。我会再次尝试，并及时反馈。

你好，我们更新了模型的modeling_chatglm.py以及github中的trans_web_demo.py，请试一下更新之后是否还有问题？

[Gongsunhuiche]

请问怎么解决的，我也是这个问题，有没有最后能run的码，第一次玩这个，求带