Skip to content

Commit

Permalink
llama : fix typical sampling (ggerganov#4261)
Browse files Browse the repository at this point in the history
Typical sampling was broken because after copying new_candidates into canditates, the "sorted" bool is left at "true", but the new data is no longer sorted according to probability. Patch to set "sorted" to false.

Test: Generating with temp=0.0001 (approx. argmax)  should generate the same sequence at typical>=1.0 and typical=0.9999 (approx. disabled, but enters the typical sampling codepath).
  • Loading branch information
tarcey authored and mounta11n committed Dec 1, 2023
1 parent e2d614f commit 8a1974a
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions llama.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -7034,6 +7034,7 @@ void llama_sample_typical(struct llama_context * ctx, llama_token_data_array * c
// Replace the data in candidates with the new_candidates data
std::copy(new_candidates.begin(), new_candidates.end(), candidates->data);
candidates->size = new_candidates.size();
candidates->sorted = false;

if (ctx) {
ctx->t_sample_us += ggml_time_us() - t_start_sample_us;
Expand Down

0 comments on commit 8a1974a

Please sign in to comment.