[Help] Correct Way to do Simple Model Constraints

[JamesKCS]

I am trying to figure out the correct way to apply simple model output constraints with transformers. As an example, let's consider how to constrain the output to a non-negative int with an unknown number of digits (decided by the model). For example, it could output 11, or 0, or 307, but not 307. or the answer is 307.

I've read https://huggingface.co/blog/constrained-beam-search; this is a great resource, but it is solving a much more challenging problem than the kind of simple model constraints that I'm looking for, and it doesn't tell us how to solve this easier problem. I'm also aware of the many 3rd party packages that accomplish this, but I want to know the correct way of doing it just using the Transformers package.

Could someone point me in the right direction (perhaps there's a blog post that I missed), or give me a code snippet of the best way to accomplish this? Perhaps I should be using something from beam_constraints or tf_logits_process? Thank you.

[zucchini-nlp]

We don't support controlled LLM generation via grammar in Transformers directly, but there are many awesome external libraries for that. You might be interested in https://github.com/outlines-dev/outlines or https://github.com/uiuc-focal-lab/syncode, as an example 🤗

[JamesKCS]

Thank you for the reply.

via grammar

I'm not looking for anything this sophisticated. For the above example, I'd be just looking to set the probabilities of all tokens except 0-9 and the EOS token to 0 (which should result in it the model being forced to output an int). If the more sophisticated constrained beam search is a possibility without external libraries, then surely this simpler case is also doable without external libraries? Like I mentioned, I need to do this without external libraries. Thank you in advance.

[zucchini-nlp]

@JamesKCS in that case you can write your own LogitsProcessor that will set all probabilities to -inf, please refer to the docs https://huggingface.co/docs/transformers/en/internal/generation_utils#logitsprocessor. You have to write something similar that suits your use case.

If you have any questions please open a discussion in the forum, we try to reserve GH for bugs and features 🤗

[JamesKCS]

Thank you, sorry for posting this in the wrong place, I'll close this (and maybe update it later with a MWE for future searchers).