Tokens returned with a GenerationResult are off when compared to text

[mattjcly]

See added printout demo.py result:

python demo.py --model ~/.cache/lm-studio/models/mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

...

token generated:  8
token generated:  4184
new text:  kings)
GenerationResult(text=' kings)', tokens=[[8, ')'], [4184, ' according']], top_logprobs=[], stop_condition=None) 

token generated:  311
new text:  according
GenerationResult(text=' according', tokens=[[311, ' to']], top_logprobs=[], stop_condition=None) 

token generated:  3230
new text:  to
GenerationResult(text=' to', tokens=[[3230, ' specific']], top_logprobs=[], stop_condition=None) 

token generated:  5718
new text:  specific
GenerationResult(text=' specific', tokens=[[5718, ' rules']], top_logprobs=[], stop_condition=None) 

token generated:  13
token generated:  128009
GenerationResult(text=' rules.', tokens=[[13, '.'], [128009, '<|eot_id|>']], top_logprobs=[], stop_condition=GenerationStopCondition(stop_reason='eos_token', stop_string='<|eot_id|>', stop_tokens=[128009]))

The tokens are essentially one token in-front of the text in the generation result. We need to align these tokens to be exactly what made up the text string