From ed7e65f5c4324a577de5b8b61784fb1876521bf8 Mon Sep 17 00:00:00 2001 From: pytorchbot Date: Wed, 9 Oct 2024 09:57:55 -0700 Subject: [PATCH] Correct Core ML perf metrics (#6000) Correct Core ML perf metrics (#5862) Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/5862 Reviewed By: huydhn Differential Revision: D63873696 Pulled By: guangy10 fbshipit-source-id: e4cb20c0a87b01d9c78dce085dad6f97bcf03b28 (cherry picked from commit 784eb51cffedb0b043462eaa9182db4d05afe04b) Co-authored-by: Guang Yang --- .github/workflows/apple-perf.yml | 2 +- extension/benchmark/README.md | 25 ++++++++++++++++++------- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/.github/workflows/apple-perf.yml b/.github/workflows/apple-perf.yml index bfa4600a98..53250f1251 100644 --- a/.github/workflows/apple-perf.yml +++ b/.github/workflows/apple-perf.yml @@ -74,7 +74,7 @@ jobs: # Separate default values from the workflow dispatch. To ensure defaults are accessible # during scheduled runs and to provide flexibility for different defaults between # on-demand and periodic benchmarking. - CRON_DEFAULT_MODELS: "stories110M,mv3,ic4,resnet50,edsr,mobilebert,w2l" + CRON_DEFAULT_MODELS: "stories110M,mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l" CRON_DEFAULT_DEVICES: "apple_iphone_15" CRON_DEFAULT_DELEGATES: "xnnpack,coreml" run: | diff --git a/extension/benchmark/README.md b/extension/benchmark/README.md index 07b8bf182c..10c7587f03 100644 --- a/extension/benchmark/README.md +++ b/extension/benchmark/README.md @@ -14,7 +14,7 @@ The ExecuTorch project introduces an advanced benchmarking infrastructure design - **Performance Monitoring**: Stores results in a database with a dashboard for tracking performance and detecting regressions. -> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time. The models performance tuning is still a work in progress. The benchmark numbers presented in this document are not vetted yet and should be treated as preliminary results. +> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time. ## Preliminary Benchmark Results @@ -27,16 +27,27 @@ Below is a table summarizing some example data points obtained via the infra. Th | DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 | | MobileNetV3 (mv3) | XNN Q8 | [34.024](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | [252.628](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461) | Samsung S22 | | MobileNetV3 (mv3) | QNN FP16 | [168.414](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | [1.182](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785) | Samsung S22 | -| MobileNetV3 (mv3) | COREML FP16 | [61.583](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | [682.197](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158) | Apple iPhone 15 Pro | -| MobileNetV2 (mv2) | XNN Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 | +| MobileNetV3 (mv3) | COREML FP16 | [57.372](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691) | [0.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691) | Apple iPhone 15 Pro | +| MobileNetV2 (mv2) | XNNPACK Q8 | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379) | Samsung S22 | | MobileNetV2 (mv2) | QNN FP16 | [136.862](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | [0.673](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097) | Samsung S22 | -| InceptionV4 (ic4) | XNN Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 | +| MobileNetV2 (mv2) | COREML FP16 | [50.892](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594) | [0.631](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594) | Apple iPhone 15 Pro | +| InceptionV4 (ic4) | XNNPACK Q8 | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716) | Samsung S22 | | InceptionV4 (ic4) | QNN FP16 | [163.943](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | [2.734](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381) | Samsung S22 | -| InceptionV3 (ic3) | XNN Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 | +| InceptionV4 (ic4) | COREML FP16 | [118.686](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140) | [4.289](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140) | Apple iPhone 15 Pro | +| InceptionV3 (ic3) | XNNPACK Q8 | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023) | Samsung S22 | | InceptionV3 (ic3) | QNN FP16 | [134.732](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | [1.351](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425) | Samsung S22 | +| InceptionV3 (ic3) | COREML FP16 | [86.728](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753) | [1.391](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753) | Apple iPhone 15 Pro | +| DeepLabV3 (dl3) | XNNPACK Q8 | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758) | Samsung S22 | +| DeepLabV3 (dl3) | QNN FP16 | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141) | Samsung S22 | +| ResNet50 (resnet50) | XNNPACK Q8 | [55.462](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | [37.021](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102) | Apple iPhone 15 Pro | +| ResNet50 (resnet50) | COREML FP16 | [68.943](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818) | [1.979](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818) | Apple iPhone 15 Pro | | TorchVisionVit (vit) | QNN FP16 | [174.430](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | [199.279](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581) | Samsung S22 | -| ResNet50 (resnet50) | COREML FP16 | [84.453](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | [1329.846](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731) | Apple iPhone 15 Pro | -| Wave2Letter (w2l) | XNN Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro | +| Wave2Letter (w2l) | XNNPACK Q8 | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319) | Apple iPhone 15 Pro | +| Wave2Letter (w2l) | COREML FP16 | [109.254](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566) | [28.465](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566) | Apple iPhone 15 | +| MobileBERT (mobilebert) | XNNPACK FP32 | [26.499](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558) | [33.978](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558) | Apple iPhone 15 Pro | +| MobileBERT (mobilebert) | COREML FP16 | [206.202](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398) | [1.873](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398) | Apple iPhone 15 Pro | +| EDSR (edsr) | XNNPACK Q8 | [3.190](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836) | [168.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836) | Apple iPhone 15 Pro | +| EDSR (edsr) | COREML FP16 | [156.075](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690) | [77.346](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690) | Apple iPhone 15 Pro | ## Supported Use Cases