Temporary PR to add to the next PR

[gsolard]

No description provided.

[maxDavid40]

New behavior to vllm (0.5.0.post1) completions endpoints (same for chat/completions) !

Usage info not return by default, need to add on the body request the stream_options informations: "stream_options" : {"include_usage": True}

Also, we need to modify benchmark_llm_serving.query_profiles.query_functions.query_functions

  if len(json_chunk['choices']) > 0:
    data = json_chunk['choices'][0]['text']
    output.generated_text += data
  if "usage" in json_chunk:
    if json_chunk['usage'] is not None:
      output.prompt_length = json_chunk['usage']['prompt_tokens']

⚠️ Be careful, the attribute stream_options not implemented in 0.4.3 in openAi_server.protocols, the response will be a 422 if you add in body the stream_options key

[gsolard]

Changes done in PR #5