-
-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature]: Allow user defined extra request args to be logged in OpenAI compatible server #5467
Comments
yeah that sounds great. We don't currently have any extra args passing via header now right? |
I don't think so but someone correct me if I am wrong. I tried passing in random extra args in the header, they don't get outputted in the logs. I also don't see an option to switch it on or off. If I were to work on this, should we just log all headers in the logs other than authorization? including model name and etc? or only the header args that have been already yet defined by vLLM? |
Given the OpenTelemetry integration has been merged. Am I still okay to proceed to work on this issue? Just checking. |
The OpenTelemetry integration uses the standard trace-id.
I think there is room to address points 2 and 3. What do you think? |
Hi @ronensc. Thanks for your response. This original feature request was just trying to allow users to add an upstream UUID in the request header, and allow vLLM to propagate it to its STDOUT so that every vLLM log entry can be easily correlated with an upstream service's request. To clarify: However, there is no way of correlating an upstream service's UUID with an individual log/request on vLLM's end. E.g. Client A has request ID 1, sends a request to vLLM where vLLM side generates an ID of 2. There currently is no easy way to link up a piece of log on vLLM's side with a request from Client A, without changing client A's code to extract vLLM's unique request ID then log it on Client A's side which is very much of an anti-pattern... Certainly happy to help out address the 2 points you mentioned tho, but maybe a slightly separate issue? Or am I confusing something...? |
@davidgxue I totally agree that it’s reasonable for vLLM to obtain a UUID from an external source and include it in its logs for tracing purposes. Additionally, I have no objections to adding custom headers to the log. |
🚀 The feature, motivation and pitch
Motivation
Using universal request ID/UUID when logging is common practice for production systems involving multiple components. My team wanted to use an UUID from upstream to trace logs produced by vLLM's OpenAI compatible webserver, but it doesn't seem like this is supported. Currently, vLLM generates its own UUID for each request, but it is 1. a separate ID, adding difficulty for tracing 2. the request ID does not always return to the API calling client, such as in cases of errors.
Solution
Allow extra args defined by the user to be passed in via a request to the OpenAI compatible frontend server, such that it can be propagated and logged via the logger.
Alternatives
"extra_args": "{ field1 : val1, field2: val2}"
) so we don't disrupt the current behavior of throwing an error when unknown request fields are passed in.Additional context
If this is approved and details are finalized, I can work on this and make a PR!
The text was updated successfully, but these errors were encountered: