Flaky behavior with grammars

[ulatekh]

I'm trying to get whisper.cpp to work with an existing grammar, i.e. to make whisper.cpp another choice for speech-engine in an existing command/control application.

After getting a lot of bad parses, I finally hacked command.exe to print the result without applying the grammar, and verified that something is going wrong inside the grammar feature. In other words, it hears what I'm saying just fine without the grammar, but when the grammar is applied, it quite often hears gibberish, e.g. text that isn't even in the grammar.

Using a grammar works fine in many cases; what bothers me is that I'm getting worse results when using a grammar. If anything, I would expect the grammar to fail by finding a valid parse, with low probability, for something that doesn't match the grammar. Instead, I'm getting results that don't match the grammar at all, and with the same vocal sample, a near-perfect match when not using the grammar at all.

Unless the grammar feature is deeply broken, I'm not sure how else to explain this. Has anyone had the same experience?

[ggerganov]

It's possible that there is a bug, but you might also have a problem in your grammar. Do the existing grammar examples work for you?

[ulatekh]

Yes, the existing grammars seem to work fine. My grammar appears to be written in the same style, yet I get inexplicable results. Here is a sample from this morning's tests:

process_general_transcription: Speech detected! Processing ...
process_general_transcription: NO GRAMMAR: txt = 'toggle radio window'
process_general_transcription:   DEBUG: txt = 'toggle ra', prob = 46.62%
process_general_transcription: Command '←[1mtoggle ra←[0m', (t = 1276 ms)

process_general_transcription: Speech detected! Processing ...
process_general_transcription: NO GRAMMAR: txt = 'pull up radio'
process_general_transcription:   DEBUG: txt = 'h', prob = 80.38%
process_general_transcription: Command '←[1mh←[0m', (t = 706 ms)

process_general_transcription: Speech detected! Processing ...
process_general_transcription: NO GRAMMAR: txt = 'toggle music window'
process_general_transcription:   DEBUG: txt = 'toggle', prob = 71.79%
process_general_transcription: Command '←[1mtoggle←[0m', (t = 1047 ms)

process_general_transcription: Speech detected! Processing ...
process_general_transcription: NO GRAMMAR: txt = 'pull up podcast'
process_general_transcription:   DEBUG: txt = 'pull up', prob = 42.74%
process_general_transcription: Command '←[1mpull up←[0m', (t = 886 ms)

process_general_transcription: Speech detected! Processing ...
process_general_transcription: NO GRAMMAR: txt = 'pull up podcast window'
process_general_transcription:   DEBUG: txt = 'pull up', prob = 81.31%
process_general_transcription: Command '←[1mpull up←[0m', (t = 1092 ms)

Attached is the grammar I'm testing with.

Note that it expects this pull request to be in use, and for cparams.fold_lowercase to be set to true. (Folding to lowercase was done to try to increase grammar recognition, by factoring out whatever case whisper.cpp felt like responding with.) It also requires this pull request, which I thought was already merged but doesn't seem to be.

My command-line for the command example was --model models\ggml-medium.bin --suppress-regex "[[:punct:]]|[ ]?[0-9]+" --grammar grammars\assistant2.gbnf --grammar-penalty 1E30 , where the model is just the standard OpenAI "medium" model converted to whisper.cpp format.

As you can see, it commonly drops the rest of the utterance after the opening word or two, and in one case, recognized "ra", which isn't in the grammar.

Any help with this would be gratefully received.

assistant2.gbnf.txt

[ulatekh]

Ugh...I believe I found the problem. My "real" grammar's top-level rule didn't start with a space. Apparently that makes all the difference in the world.