[New Model]: Support Nemotron-4-340B

[dskhudia]

🚀 The feature, motivation and pitch

Benchmarks on the Nvidia's latest nemotron model look great. Is there plan or already going work to support it?

Alternatives

No response

Additional context

https://huggingface.co/nvidia/Nemotron-4-340B-Instruct

[mgoin]

Once there is support in HF transformers, then it should be relatively straightforward to port into vLLM. It seems there aren't any efforts from searching the transformers issues/PRs

[riverind]

attention

[natolambert]

I started a paid bounty to close these issues. Already over $200 of support.
https://x.com/natolambert/status/1814735390877884823