Skip to content

03. Usage

Jump to bottom
Brian Dashore edited this page Sep 1, 2024 · 2 revisions

Usage

TabbyAPI's main use-case is to be an API server for running ExllamaV2 models.

API Server

Currently TabbyAPI supports clients that use the OpenAI standard and KoboldAI's API.

In addition, there are expanded parameters to generation endpoints along with administrative endpoints for loading, unloading, loras, sampling overrides, etc.

Note

If you are a developer and want to add full TabbyAPI support to your app, it's recommended to use the autogenerated documentation.

Below is an example CURL request using the OpenAI completions endpoint:

curl http://localhost:5000/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "meta-llama/Meta-Llama-3-8B",
  "prompt": "Once upon a time,",
  "max_tokens": 400,
  "stream": false,
  "min_p": 0.05,
  "repetition_penalty": 1.05
}'

Authentication

Every call to a TabbyAPI endpoint requires some form of authentication. Keys have two types of permissions:

  • API: Accesses non-invasive endpoints (ex. generation, model list fetching)
  • Admin: Allowed to access protected endpoints that deal with resources (ex. loading, unloading)

In addition, when calling list endpoints, API keys will only fetch the currently loaded object while admin keys will list the entire directory. For example, calling /v1/models will return a list of the user-configured models directory only if an admin key is passed.

Therefore, it's recommended to keep the admin key for yourself and only share the api key with users.

If these keys get compromised, shut down your server, delete the api_tokens.yml file, and restart. This will generate new keys which you can share with users.

To bypass authentication checks, set disable_auth to True in config.yml. However, turning off authentication without a third-party solution will make your instance open to the world.

Difficult to get started?

Is the API difficult? Don't want to load models with config.yml? That's okay! Not everyone is a master user of AI products when starting out.

For newer users, it's recommended to use a UI that allows for managing TabbyAPI via API endpoints.

To find UI projects, take a look at Community Projects for more information.

The Discord is also a great place to ask for help. Please be nice when asking questions as all the developers are volunteers who have lives outside of TabbyAPI.

Clone this wiki locally