Exploit vLLM options to return deltas/final-output only

[njhill]

Nontrivial performance benefit, particularly when running with decoupled front-end process.

These changes require vLLM >= 0.6.1.post2

[njhill]

@dtrifiro if necessary I could update these changes to work both before and after v0.6.1.post2 (like you've done for other things)

[dtrifiro]

@njhill yeah I think that'd be better, we can bump the minimum vllm version and drop the backward compatibility code after the next vllm version is out

[njhill]

Now that vllm 0.6.2 is released, @dtrifiro agreed that there's no need for the aforementioned backwards compatibility changes, so this PR should now be ready to merge.

[dtrifiro]

@njhill it seems this broke streaming generation

E                   	status = StatusCode.UNKNOWN
E                   	details = "Unexpected <class 'TypeError'>: object of type 'NoneType' has no len()"
E                   	debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:45943 {created_time:"2024-09-27T00:19:45.777203684+00:00", grpc_status:2, grpc_message:"Unexpected <class \'TypeError\'>: object of type \'NoneType\' has no len()"}"
E                   >

See traceback here: https://github.com/opendatahub-io/vllm-tgis-adapter/actions/runs/11062189283/job/30736218398?pr=137#step:8:2614

This can be merged after #143

edit: the CI failure could be due to this branch not being updated, can you rebase on #143 ?

edit: Already rebased, still failing

[njhill]

@dtrifiro this one is now fixed and should be working

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 62.96296% with 10 lines in your changes missing coverage. Please review.

Project coverage is 58.35%. Comparing base (dd0eb1c) to head (fb849d9).

Files with missing lines	Patch %	Lines
src/vllm_tgis_adapter/grpc/grpc_server.py	62.50%	3 Missing and 6 partials ⚠️
src/vllm_tgis_adapter/utils.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #137   +/-   ##
=======================================
  Coverage   58.34%   58.35%           
=======================================
  Files          27       27           
  Lines        1611     1616    +5     
  Branches      268      270    +2     
=======================================
+ Hits          940      943    +3     
+ Misses        582      581    -1     
- Partials       89       92    +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[njhill]

Remaining test failures appeared to be unrelated to this PR.

Probably due to other upstream changes, and we still need to investigate/address of course.

[njhill]

Yeah looks like those are due to an upstream bug with CPU backend vllm-project/vllm#9024.