Performance

Performance

Framework	Version	Architecture	Archive	Mode	HTML	XLSX	Comments
Intel® Distribution of OpenVINO™ Toolkit	2023.3	x86	ZIP TAR.GZ	Host	LINK	XLSX	Python API Metrics: FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
TFLite	2.14.0	RISC-V	ZIP TAR.GZ	Host	—	XLSX	Python API Metrics: FPS Parameters: Models downloaded and converted to .json by script convert_models_to_tflite.sh
Apache TVM	0.15	RISC-V	ZIP TAR.GZ	Host	—	XLSX	Python API Metrics: FPS Parameters: Models downloaded and converted to .json by script convert_models_to_tvm.sh and compiled to .tar by script compile_tvm_models.sh
Apache TVM	0.14.dev264	x86	ZIP TAR.GZ	Host	—	XLSX	Python API Metrics: FPS Parameters: Models downloaded and converted to .json by script: convert_models_to_tvm.sh
Apache TVM	0.14.dev264	x86	ZIP TAR.GZ	Docker	—	XLSX	Python API Metrics: FPS Parameters: Models downloaded and converted to .json by script: convert_models_to_tvm.sh
Intel® Distribution of OpenVINO™ Toolkit	2023.0	x86	ZIP TAR.GZ	Host	LINK	XLSX	Python API Metrics: FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
MXNet	1.9.1	x86	ZIP TAR.GZ	Host	—	XLSX	Python API Metrics: FPS
MXNet	1.9.1	x86	ZIP TAR.GZ	Docker	—	XLSX	Python API Metrics: FPS
ONNX Runtime	1.14.0	x86	ZIP TAR.GZ	Host	—	XLSX	Python API Metrics: FPS
ONNX Runtime	1.14.0	x86	ZIP TAR.GZ	Docker	—	XLSX	Python API Metrics: FPS
Intel® Distribution of OpenVINO™ Toolkit	2022.3	x86	ZIP TAR.GZ	Docker	LINK	XLSX	Python API Metrics: FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
Intel® Distribution of OpenVINO™ Toolkit	2022.2	x86	ZIP TAR.GZ	Docker	LINK	XLSX	Python API Metrics: Batch FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
Intel® Distribution of OpenVINO™ Toolkit	2022.2	x86	ZIP TAR.GZ	Host	LINK	XLSX	Python API Metrics: Batch FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
Intel® Distribution of OpenVINO™ Toolkit	2022.1	x86	ZIP TAR.GZ	Docker	LINK	XLSX	Python API Metrics: Batch FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
Intel® Distribution of OpenVINO™ Toolkit	2021.4	x86	ZIP TAR.GZ	Docker	LINK	XLSX	Python API Metrics: Batch FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
Intel® Distribution of OpenVINO™ Toolkit	2020.2	x86	ZIP TAR.GZ	Docker	Public Models Intel Models	—	Python API Metrics: Batch FPS Parameters: Default number of threads in the latency mode Optimal number of streams in the throughput mode, number of streams equals number of requests
Intel® Distribution of OpenVINO™ Toolkit	R3	x86	ZIP TAR.GZ	Docker	LINK	—	Python API Metrics: Batch FPS Parameters: Default number of threads in the sync mode Optimal number of streams in the async mode, number of streams equals number of requests
Intel® Distribution of OpenVINO™ Toolkit	R5.01	x86	ZIP TAR.GZ	Host	LINK	—	Python API Metrics: Latency (for the sync mode) and average time of the single pass (for the async mode) Batch FPS Parameters: Number of threads in the sync mode is written in the table header ("Thread count") Number of requests and streams in the async mode is written in the table header ("Async request count" and "Stream count")

Note: DLI supports two execution modes. The first one is an execution in Docker container (it is named Docker in the table), and the second one is a direct execution on the host (it is named Host in the table).