About gemma-2-9b-it AWQ quantization

[fresh-minari]

Hi, I saw the AWQ model of gemma-2-9b-it on HF (solidrust/gemma-2-9b-it-AWQ) and it works well on vLLM. However, when I tried to quantize same model (google/gemma-2-9b-it) with the code you uploaded on github, the outputs doesn't work well. I've got this error when I validate the quantized model.

'Gemma2LikeModel' object has no attribute '_prepare_4d_causal_attention_mask_with_cache_position'

how can I solve it?

[suparious]

Sorry for the late reply, but the Gemma models have really been slow to support.
I think the issue would of been solved from the upstream development in AutoAWQ.

The reason why my version was working, is because I painstakingly compiled my own version of AutoAWQ from the master branch, with and updated version of every dependency and library (I was obsessed).