decoder_input_details and return_full_text parameters stopped providing previous log probs

[sioan]

System Info

In version 2.4.1 sending an http request with parameters return_full_text and decoder_input_details set to true provided log_probs of all input tokens.

By version 3.1.0, return_full_text no longer works and only the new token and it's log prob is return

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

given a running tgi v3.1.0 instance, the following python code reproduces the effect

import json
import requests
url="http://localhost:8000"
prompt = "San Francisco is a"
payload = {"inputs":prompt, "parameters":{"max_new_tokens":1, "temperature":0.5,"return_full_text":True,"details":True,"decoded_input_details":True}}
r = requests.post(url,json=payload)

print(r.json())

I'll update with more info as I debug further.

Expected behavior

Expect the results from above to satisfy

assert len(r.json()[0]['details']['prefill'] )> 1