Discussion: Architecture for Jan to support multiple Inference Engines

[dan-menlo]

Objective

• Jan's architecture will default to Nitro, but have flexibility to incorporate other Model Backends / Inference Engines
• We see very fast movement around a few ecosystems
  • Intel BigDL
  • Intel Extensions for Transformers
  • TensorRT-LLM for WIndows
  • And many more to come
• Jan's Architecture needs to be able to incorporate modular Inference Engines
  • Parallelize efforts to support different inference engines
  • Allow us to hedge technical architectural risks

Solution

I envision an architecture in Jan that has the following:

• Model Extension
• Inference Extension
• [Many] Extensions for each Inference Engine

Models Extension

• Handles Models filesystem and CRUD
• /models API endpoint

Inference Extension

• Handles generic endpoints (for now, /chat/completions, later /audio/speech)
• Routes an inference request to the correct Inference Engine extension (defined by model.json)

Extension for each Inference Engine

• Each Inference Engine exposes a /chat/completions endpoint
• Each Inference Engine is expected to have its own OpenAI compliant API
• Each Inference Engine is expected to do its own automatic resource management (e.g. loading, unloading)
• Each Inference Engine is expected to stream events to the Frontend UI
  • e.g. Model Loading, Unloading events (i.e. "please wait")
  • e.g. SSEs for model response
• Extensions for Inference Engine are available via The Hub
  • These are NPM packages that build on our Extensions API

Example

File Tree

/jan
    /models
        /llama2-70b
             llama2-gguf-q4_k_m.bin     # uses Nitro
             model.json
        /llama2-70b-intel-bigdl
              #pytorch files
              model.json
    /engines
        /nitro
            engine.json
        /openai

model.json gpt4-32k-1603

{
    engine: "openai"    // If not specified, defaults to Nitro
}

engine.json example for Nitro

{
    settings: { ...default_gguf_settings},
    parameters: { ...default_gguf_params},
}

Execution Path

1. User makes inference request to llama2-70b-intel-bigdl
2. Inference Extension loads the model.json for llama2-70b-intel-bigdl and sees engine is intel-bigdl
3. Inference Extension routes it to intel-bigdl Inference Engine Extension
4. intel-bigdl Inference Engine Extension takes in /chat/completions request, runs inference, and returns result through SSE

[hiro-v]

1. Idk what models extension do when we have flat and separate model extensions
2. Nitro and anything as remote models (oai/ azure oai/ claude/ tgi compatible) will be supported by Dec 15
3. I will check if the nvidia local llm support or not but this is optional, but as in their announcement the 0.6.0 will be out in 2023 Dec.

[freelerobot]

Nitro is still an intermediary server to llama.cpp. It is opinionated, uses Drogon cpp, and is the default engine when users don't specify the engine for their GGUF models.

[dan-menlo]

@vuonghoainam Re supporting Claude/TGI, aren't there differences between their interfaces (vs. OAI)? Does this mean we will need separate inference engine extensions for each of them?

I can also see how it would not be a huge amount of work, just some Javascript glueing.

• Claude: https://docs.anthropic.com/claude/reference/complete_post
• TGI: https://huggingface.co/docs/text-generation-inference/basic_tutorials/consuming_tgi

[hiro-v]

Yes, @dan-jan it's correct that we add the support for TGI and claude like that.

Some decisions:

• Change in model.json in https://github.com/janhq/jan/tree/main/plugins/model-plugin
  ===> Will update engine as enum, and this is the field that inference engines will pick up to use
  ===> For remote models: The settings are mostly secrets that will be supported by combining with this: feat: Extension should be able to register its own settings #776 to persist
• The inference-engine from https://github.com/janhq/jan/tree/main/plugins/inference-plugin will be refactored as follow and each one will be published as @janhq/inference-engine-<name>.

inference-engine/
    nitro/ (default)
    openai/
    azure-openai/
    claude/
    hf-endpoints/
    nvidia-triton-trt-llm/

• On the right hand side, the parameters will be there for thread settings to update.

[dan-menlo]

@vuonghoainam This looks quite clear. I'll create a task in the epic to track documenting this in Specs.

[freelerobot]

Moving to #1271