Allow reusing results from llama_token_to_piece when sampling grammars

[MarcusDunn]

This is an amendment to #4210 which allows passing in already computed (ideally shared across calls) strings to avoid calling llama_token_to_piece in llama_sample_grammar. Behavoir remains unchanged by passing in nullptr.

See #4210 for the "before".

I'm marking as a draft for now to allow #4210 to be merged in and then I'll clean up the git history.

Also I wanted to see if this is worth a breaking API change (if it is - I'll go fix up other spots in the code where the change can be taken advantage of.)

[cebtenzzre]

The doc comment for this function doesn't clearly state that individual pieces may be nullptr.

[MarcusDunn]

This is mostly playing defence as std::string cannot handle nullptrs / empty c strings (from my understanding - I'm a complete c++ novice)

The lack of docs allows swapping the behavior from calling token_to_piece to assuming a null entry is an empty string, but from my measurements the performance impact was negligible.

If we want to document the current behavior that's fine by me.

[cebtenzzre]

std::string can represent empty strings, but not null. If it was a C string you would still have to do something with it eventually if it is null to prevent a segfault.

Since this code is apparently not used yet, it's hard to reason about performance or what the correct inputs should be.

[ggerganov]

Also I wanted to see if this is worth a breaking API change (if it is - I'll go fix up other spots in the code where the change can be taken advantage of.)

Wouldn't it be better if we pre-compute the pieces when the model is loaded and store it in llama_vocab?

llama.cpp/llama.cpp

Lines 1329 to 1332 in 3e73d31

	std::unordered_map<token, id> token_to_id;
	std::vector<token_data> id_to_token;

This way the user does not need to do it themselves.

[MarcusDunn]

Closing in favor of implementing #4213 (comment)

Done in #4330