Supprot for qwen2moe gguf models

[BrenchCC]

System Info

• transformers version: 4.45.0.dev0
• Platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.35
• Python version: 3.10.14
• Huggingface_hub version: 0.24.5
• Safetensors version: 0.4.4
• Accelerate version: 0.33.0
• Accelerate config: - compute_environment: LOCAL_MACHINE
  - distributed_type: DEEPSPEED
  - use_cpu: False
  - debug: True
  - num_processes: 24
  - machine_rank: 2
  - num_machines: 3
  - main_process_ip: gpu007
  - main_process_port: 9901
  - rdzv_backend: static
  - same_network: True
  - main_training_function: main
  - enable_cpu_affinity: False
  - deepspeed_config: {'deepspeed_config_file': '/data/vayu/train/config/deepspeed/zero2.json', 'deepspeed_hostfile': '/data/vayu/train/config/hostfile', 'deepspeed_multinode_launcher': 'pdsh', 'zero3_init_flag': False}
  - downcast_bf16: no
  - tpu_use_cluster: False
  - tpu_use_sudo: False
  - tpu_env: []
• PyTorch version (GPU?): 2.4.0 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?:
• Using GPU in script?:
• GPU type: NVIDIA A800-SXM4-80GB

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoModelForCausalLM

model_id = "./qwen2moe_4x1.5b/"
file_name = "Qwen2-4x1.5B-reasoning-pro-Q4_K_M.gguf" # local file, base on Qwen/Qwen2-1.5B-Instruct

model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

Expected behavior

miniconda3/envs/vllm_cu12/lib/python3.10/site-packages/transformers/modeling_gguf_pytorch_utils.py", line 100, in load_gguf_checkpoint
raise ValueError(f"Architecture {architecture} not supported")
ValueError: Architecture qwen2moe not supported

[LysandreJik]

cc @SunMarc :)

[LysandreJik]

or @julien-c 👀

[SunMarc]

Hey @BrenchCC ! Thanks for raising this issue ! I will create a community contribution issue, so we get faster support for gguf models such as qwen2moe.

[SunMarc]

Opened this #33260 !