Interface with text-generation-inference (TGI)

[rlouf]

From my understanding of the code in TGI (building the FSM and using the FSM to bias logits) and discussions on the hf-dottxt-outlines-collab TGI interacts with Outlines in the following way:

1. The FSM is compiled in Python using RegexFSM
2. The instance of RegexFSM is used to bias logits

While (2) can remain in Python we would like to port the logic necessary to build the FSM (1) to Rust. This means porting:

• create_fsm_index_tokenizer;
• The interegular library used to build a DFA from a regular expression;
• The JSON Schema to Regex translation logic;
• Ideally we can integrate with the Rust tokenizers directly so we don't have to deal with type conversion.

[rlouf]

If we need extra work to be done for this it would be better to open specific issues. Closing for now.