Files

inference

Name		Name	Last commit message	Last commit date
parent directory ..
ascend/mindformers		ascend/mindformers
faster-transformer		faster-transformer
flexflow-serve		flexflow-serve
hf-transformer		hf-transformer
tensorrt-llm		tensorrt-llm
tensorrt		tensorrt
triton		triton
vllm		vllm
FlexFlow-Serve.md		FlexFlow-Serve.md
README.md		README.md
TensorRT-LLM.md		TensorRT-LLM.md
vllm.md		vllm.md
推理性能评估指标.md		推理性能评估指标.md

README.md

LLM 推理

OpenLLM: An open platform for operating large language models (LLMs) in production.
Triton
Text Generation Inference：https://github.com/huggingface/text-generation-inference
vLLM：https://github.com/vllm-project/vllm
FastTransformer：https://github.com/NVIDIA/FasterTransformer
vLLM：https://github.com/vllm-project/vllm
LLM Accelerator：https://github.com/microsoft/LMOps
LMDeploy,TurboMind
BigDL：https://github.com/intel-analytics/BigDL
OpenVINO：https://github.com/openvinotoolkit/openvino
AWQ、AutoAWQ