[Core][Model] Add simple_model_runner and a new model XLMRobertaForSequenceClassification through multimodal interface

[AllenDou]

This PR,

1. Add a new model XLMRobertaForSequenceClassification (for RAG scenario) https://github.com/huggingface/transformers/blob/v4.42.3/src/transformers/models/xlm_roberta/modeling_xlm_roberta.py
   which processes input data through a multimodal interface like the following

outputs = llm.process([{
    "prompt": prompt,
    "multi_modal_data": {
        "xlmroberta": inputs,
    }
}])

2. Introduce class ModelMode [DECODER, ENCODER, ENCODER_DECODER, EMBEDDING, SIMPLE]
3. Add a new model runner(simple_model_runner.py), I didn't use model_runner.py or embedding_model_runner.py because they have procedures like logits(), sample(), and pooling() that we don't need. Maybe we could find a way to create a more general model runner to support 'any' model
4. Add SimpleSequenceGroupOutput, SimpleRequestOutput, and SimpleOutput for general output
5. Tensor-parallel is enabled, but no benefits have been achieved so far
6. Test case/example added

I have two goals:

1. To find a way to enable vllm to integrate 'any' model
2. To replace these models' layers with vllm/layers/* as well as TP/PP to achieve performance improvement

CLOSE #6424
CLOSE #6789

[DarkLight1337]

If I'm understanding this PR correctly, you are basically using the multi-modal interface to pass data directly to the model (in this case the input IDs and attention mask). Are you working towards making vLLM function out-of-the-box with generic HuggingFace models?

[DarkLight1337]

Left some initial comments.

[AllenDou]

If I'm understanding this PR correctly, you are basically using the multi-modal interface to pass data directly to the model (in this case the input IDs and attention mask). Are you working towards making vLLM function out-of-the-box with generic HuggingFace models?

Yes, you are right. In fact, I have two goals:

1. To find a way to enable vllm to integrate 'any' model.
2. To replace these models' layers with vllm/layers/* as well as TP/PP to achieve performance improvement.

After I replaced XLMRobertaForSequenceClassification's query/key/value linear with QKVParallelLinear, I saw a 15% performance improvement.

[DarkLight1337]

Added some suggestions to improve the type annotations.

[DarkLight1337]

The config and multi-modal parts LGTM. I assume the model is implemented correctly since the tests pass.

However, since I'm not involved with the internals of model executor, block manager and workers, I'll leave it to @robertgshaw2-neuralmagic to review those. He will also see how to integrate this into the existing work for encoder-decoder support.

[robertgshaw2-neuralmagic]

Im going to take a look at this over the weekend. Thanks @AllenDou!

[AllenDou]

mark #6424

[AllenDou]

/ready

[AllenDou]

Hello @robertgshaw2-neuralmagic , just a friendly reminder to review this PR when you get a chance,
We want to enable more models through this feature, thank a lot!

[robertgshaw2-neuralmagic]

Thanks @AllenDou ! this is on my list

[AllenDou]

mark #6789

[AllenDou]

mark #1187 #205

[DarkLight1337]

I think @robertgshaw2-neuralmagic has been very busy lately... Are you still interested in implementing this? Let's split the PR into smaller parts that are more manageable so that @robertgshaw2-neuralmagic doesn't have to review everything at once.

(By the way, please take into account the changes from #4942.)

[AllenDou]

I think @robertgshaw2-neuralmagic has been very busy lately... Are you still interested in implementing this? Let's split the PR into smaller parts that are more manageable so that @robertgshaw2-neuralmagic doesn't have to review everything at once.

(By the way, please take into account the changes from #4942.)

Of course, after reviewing #4942, I realized that nm folks have their own methods for supporting 'anymodel'. So, I think maybe I should close this PR. If this feature is needed in the future, I can reopen this PR at any time.

[robertgshaw2-neuralmagic]

Hey @AllenDou Im really sorry for the delay here, have had no bandwidth over the past month

#4942 is only for encoder-decoder, which is separate from the feature here, so I do not think this work is a duplicate

I’m happy to pick this back up if you want to reopen the PR

[AllenDou]

Hey @AllenDou Im really sorry for the delay here, have had no bandwidth over the past month

#4942 is only for encoder-decoder, which is separate from the feature here, so I do not think this work is a duplicate

I’m happy to pick this back up if you want to reopen the PR

Great, I'll resolve the conflicts and reopen the PR, I hope this won't take up too much of your time :)

[robertgshaw2-neuralmagic]

Thanks so much, I really appreciate your patience. Resolving our performance issues has just become a huge priority

[AllenDou]

Hi @robertgshaw2-neuralmagic , I have rebased and reopened this PR. Please take a look at it. The offline_inference_xlmroberta_awq.py file is a quantized version I've been working on, but I'm encountering some technical difficulties that are proving challenging to overcome. So, this file could be removed if necessary. cc @DarkLight1337

[AllenDou]

offline_inference_xlmroberta_awq.py is deleted, as after hacking autoawq for xlmroberta model, I see no gain of performance benifit under vllm serving. Trying to fp8.

[prashil1996]

@AllenDou @robertgshaw2-neuralmagic @DarkLight1337
Is there any timeline on when this could be delivered?

[AllenDou]

@AllenDou @robertgshaw2-neuralmagic @DarkLight1337 Is there any timeline on when this could be delivered?

Maybe we should wait until @robertgshaw2-neuralmagic gets a chance to review this PR.

[DarkLight1337]

A quick heads-up that the new locations of the model tests have been adjusted in #7820, so please merge from main.

[DarkLight1337]

Also @robertgshaw2-neuralmagic do you have any timeframe on when you will be available to review this PR?

[fan-niu]

@AllenDou Hello, thank you for your contribution. I use concurrency to request services, and there will be a series of error reports, including the following errors:
File "/share5/projects/llm/finetune-accelerate/inference/tensorrtllm-experiment/sglang/code/vllm_test/vllm_sqclassification/vllm/vllm/model_executor/models/xlmroberta.py", line 721, in forward batch_size, seq_length = input_shape ValueError: not enough values to unpack (expected 2, got 1)

Can you please help to optimize async requests? Thanks

Note:
I use http://127.0.0.1:8081/v1/completions, concurrency = 2,
payload:
{
"prompt": "what time is it",
"model":MODEL_NAME,
"stop": "<\s>",
"max_tokens":128,
"stream": False,
"presence_penalty": 0,
"frequency_penalty": 0,
"temperature": 0.0,
"top_p": 1,
}

[AllenDou]

@AllenDou Hello, thank you for your contribution. I use concurrency to request services, and there will be a series of error reports, including the following errors: File "/share5/projects/llm/finetune-accelerate/inference/tensorrtllm-experiment/sglang/code/vllm_test/vllm_sqclassification/vllm/vllm/model_executor/models/xlmroberta.py", line 721, in forward batch_size, seq_length = input_shape ValueError: not enough values to unpack (expected 2, got 1)

Can you please help to optimize async requests? Thanks

Note: I use http://127.0.0.1:8081/v1/completions, concurrency = 2, payload: { "prompt": "what time is it", "model":MODEL_NAME, "stop": "<\s>", "max_tokens":128, "stream": False, "presence_penalty": 0, "frequency_penalty": 0, "temperature": 0.0, "top_p": 1, }

This PR currently does not support frontend access through the http protocol. By the way, the XLM-Roberta model compares the similarity of two strings. Therefore, you need to pass a tuple containing two strings (string, string) as input. Please refer to examples/offline_inference_xlmroberta.py for more details.

[fan-niu]

@AllenDou Thanks for your reply, can you tell me how to add support for http requests on your branch? Thank you, or can you add this function if it is convenient for you? My final goal is to need llamaForSequenceClassification based on http request, I have added llamaForSequenceClassification to llama.py with reference to your branch, and implemented the forward function, but currently I cannot make the correct request through http.