Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.

Run inference on MPT-30B using CPU

Self-evaluating interview for AI coders

Chat with your data privately using MPT-30b

Run inference on replit-3B code instruct model using CPU

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

This custom_node for ComfyUI adds one-click "Virtual VRAM" for any GGUF UNet and CLIP loader, managing the offload of layers to DRAM or VRAM to maximize the latent space of your card. Also includes nodes for directly loading entire components (UNet, CLIP, VAE) onto the device you choose. Includes 16 examples covering common use cases.

This repository contains a web application designed to execute relatively compact, locally-operated Large Language Models (LLMs).

stable-diffusion.cpp bindings for python

Finetuning instruct-LLaMA on german datasets.

✨ Your Custom Offline Role Play with LLM and Stable Diffusion on Mac and Linux (for now) 🧙‍♂️

PyGPTPrompt: A CLI tool that manages context windows for AI models, facilitating user interaction and data ingestion for optimized long-term memory and task automation.

Some random tools for working with the GGUF file format

convert a saved pytorch model to gguf and generate as much corresponding ggml c code as possible

A simple chat app with embeddings and vector database, exclusively for local execution on mac/apple silicon.

✨ LLM CMD is a toolbox allowing you to use LLM in daily developer commands 💻

finLLM using llama-2-7b-chat-ggml, langchain, chainlit, and faiss vector db on encyclopedia of finance, sfsu

Development and deployment on AWS of a question-answer LLM model using Llama2 with 7B parameters and RAG with LangChain

An AI_Podcast developed with running Xinference platform using GGML and Pytorch Model

An LLM App that generates Blogs based on style, topic and number of words input by the user.