Cannot use vLLM to serve the model

[huaiyizhao]

The demo_vllm.py runs smothly but vllm serve ./ckpt --trust-remote-code does not work.

vllm==0.8.5
flash-attn==2.7.4.post1
torch==2.6.0

[xiez22]

Please try this: https://github.com/xiez22/vllm/tree/timeseries

[xiez22]

Follow this doc to install the vllm (https://github.com/xiez22/vllm/tree/timeseries) from source: https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-wheels

Remember to switch to the timeseries branch before installation!

[xiez22]

Example API usage after installation:

import openai
import numpy as np

# Load Time Series Data
SEQ_LEN_1 = 256
SEQ_LEN_2 = 1000

x1 = np.arange(SEQ_LEN_1)
x2 = np.arange(SEQ_LEN_2)

# TS1: A simple sin signal with a sudden decrease
ts1 = np.sin(x1 / 10) * 5.0
ts1[103:] -= 10.0

# TS2: A increasing trend with a upward spike
ts2 = x2 * 0.01
ts2[100] += 10.0
prompt = f"I have 2 time series. TS1 is of length {SEQ_LEN_1}: <ts><ts/>; TS2 if of length {SEQ_LEN_2}: <ts><ts/>. Please analyze the local changes in these time series first and then conclude if these time series showing local changes near the same time?"
prompt = f"<|im_start|>system\nYou are a helpful assistant.<|im_end|><|im_start|>user\n{prompt}<|im_end|><|im_start|>assistant\n"

# Create ts_list
ts_list = [ts1.tolist(), ts2.tolist()]

# Send message
client = openai.OpenAI(base_url="http://127.0.0.1:12345/v1", api_key="test")

response = client.chat.completions.create(
    model="chatts",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": prompt}
            ] + [{"timeseries": ts} for ts in ts_list],
        }
    ]
)

print(response.choices[0].message.content)

[Adonishl]

Hi, @xiez22 I have tried to rebuild vllm and it seems the server is normally launched. But when I use the API Example as client, it came up with 500 ERROR. The server log is:

If you ever see the problem like this? Should I launch the server with any chat template?

[xiez22]

Please try using the following code to launch the server:

vllm serve ./ChatTS-14B \
    --served-model-name chatts \
    --trust-remote-code \
    --hf-overrides '{"model_type":"chatts"}' \
    --max-model-len 6000 \
    --gpu-memory-utilization 0.97 \
    --limit-mm-per-prompt timeseries=15 \
    --allowed-local-media-path $(pwd) \
    --host 0.0.0.0 \
    --port 12345 \
    --uvicorn-log-level debug

[xiez22]

@Adonishl @huaiyizhao Does this solve your problem?

[Adonishl]

@Adonishl @huaiyizhao Does this solve your problem?

I found that the 500 ERROR is solved by setting VLLM_ALLOW_INSECURE_SERIALIZATION=1. Thanks