Make number of generated tokens consistent with CLI

[rasbt]

Updated the llm.benchmark() method to make the number of generated tokens used for the calculation consistent with those in the CLI via litgpt generate and litgpt chat.

[rasbt]

Actually, I don't think the stream=False vs stream=True difference is an issue in the LLM API. It's reproducible for the LitGPT CLI as well. @Andrei-Aksionov @apaz-cli

Here are screenshots to show that I am not making this up.

1) Stream=False

• For a fair comparison, you need to set this to False:

litgpt/litgpt/generate/base.py

Line 92 in 1bd8600

include_prompt: bool = True,

2) Stream=True

TLDR

So if we see this difference between streaming and non-streaming, I think this is a legacy difference, not something introduced by the LLM API or the benchmark code. Maybe the unified code in #1675 will address this. Otherwise, this is a future issue to look into.

[Andrei-Aksionov]

Interesting. Thanks for the comparison @rasbt
That would an interesting task to solve. Maybe one day.

Or perhaps @apaz-cli might have noticed something regarding this issue while working on his PR.

[apaz-cli]

@rasbt I tested both streaming and nonstreaming against the new impl. They both definitely call next_token() the same number of times, with the same args. It also makes the same call to cat().

Literally the only difference is

# Chat
# Scalarize the tensor for all iterations beyond the first
input_pos[-1:].add_(1)

vs

# Generate, new impl
# Post-prefill create scalar tensor directly
input_pos = input_pos.add_(1)

But this being significant doesn't line up with the theory. Don't know.