dynamic-batching

Here are 3 public repositories matching this topic...

【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接口方式提供服务。

tensorflow torch tensorrt serving triton-inference-server dynamic-batching vllm tensorrt-llm

Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.

python deep-learning inference gpt performance-optimization dynamic-batching llm

InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.

Add a description, image, and links to the dynamic-batching topic page so that developers can more easily learn about it.

To associate your repository with the dynamic-batching topic, visit your repo's landing page and select "manage topics."