You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The generation method in hf_transformers.py is decorated with outlines.vectorize. Thus when we pass, say a 2-dimensional array, we run the model twice when it would be more efficient to flatten the input array and perform batch inference. We thus need to implement a mechanism that allows to flatten the array before passing it to the decorated function.
To do this, we have a choice between implementing a new decorator (preferred) or change the behavior of outlines.vectorize.
The text was updated successfully, but these errors were encountered:
The generation method in
hf_transformers.py
is decorated withoutlines.vectorize
. Thus when we pass, say a 2-dimensional array, we run the model twice when it would be more efficient to flatten the input array and perform batch inference. We thus need to implement a mechanism that allows to flatten the array before passing it to the decorated function.To do this, we have a choice between implementing a new decorator (preferred) or change the behavior of
outlines.vectorize
.The text was updated successfully, but these errors were encountered: