Pinned Loading

1. DistServe DistServe Public

   Forked from LLMServe/DistServe

   Disaggregated serving system for Large Language Models (LLMs).

   Jupyter Notebook

2. FlexFlow FlexFlow Public

   Forked from flexflow/flexflow-train

   FlexFlow Serve: Low-Latency, High-Performance LLM Serving

   C++

3. INFaaS INFaaS Public

   Forked from stanford-mast/INFaaS

   Model-less Inference Serving

   C++

4. Mooncake Mooncake Public

   Forked from kvcache-ai/Mooncake

   Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

   C++

5. Quest Quest Public

   Forked from mit-han-lab/Quest

   [ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

   Cuda

6. vllm vllm Public

   Forked from vllm-project/vllm

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python