Correct Core ML perf metrics (#5862)

Summary: Pull Request resolved: #5862 Reviewed By: huydhn Differential Revision: D63873696 Pulled By: guangy10 fbshipit-source-id: e4cb20c0a87b01d9c78dce085dad6f97bcf03b28
pytorch · Oct 4, 2024 · 784eb51 · 784eb51
1 parent 34e7ad8
commit 784eb51
Show file tree

Hide file tree

Showing 2 changed files with 12 additions and 7 deletions.
diff --git a/.github/workflows/apple-perf.yml b/.github/workflows/apple-perf.yml
@@ -74,7 +74,7 @@ jobs:
           # Separate default values from the workflow dispatch. To ensure defaults are accessible
           # during scheduled runs and to provide flexibility for different defaults between
           # on-demand and periodic benchmarking.
-          CRON_DEFAULT_MODELS: "stories110M,mv3,ic4,resnet50,edsr,mobilebert,w2l"
+          CRON_DEFAULT_MODELS: "stories110M,mv3,mv2,ic4,ic3,resnet50,edsr,mobilebert,w2l"
           CRON_DEFAULT_DEVICES: "apple_iphone_15"
           CRON_DEFAULT_DELEGATES: "xnnpack,coreml"
         run: |

diff --git a/extension/benchmark/README.md b/extension/benchmark/README.md
@@ -14,7 +14,7 @@ The ExecuTorch project introduces an advanced benchmarking infrastructure design
 
 - **Performance Monitoring**: Stores results in a database with a dashboard for tracking performance and detecting regressions.
 
-> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time. The models performance tuning is still a work in progress. The benchmark numbers presented in this document are not vetted yet and should be treated as preliminary results.
+> **Disclaimer:** The infrastructure is new and experimental. We're working on improving its accessibility and stability over time.
 
 
 ## Preliminary Benchmark Results
@@ -25,22 +25,27 @@ Below is a table summarizing some example data points obtained via the infra. Th
 |-----------------------|--------------|----------------------|----------------------------|------------------------|
 | MobileNetV3 (mv3)     | XNNPACK Q8   | [34.024](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461)               | [252.628](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218988461)                    | Samsung S22            |
 | MobileNetV3 (mv3)     | QNN FP16     | [168.414](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785)              | [1.182](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987785)                      | Samsung S22            |
-| MobileNetV3 (mv3)     | COREML FP16  | [61.583](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158)               | [682.197](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078681158)                    | Apple iPhone 15 Pro    |
+| MobileNetV3 (mv3)     | COREML FP16  | [57.372](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691)               | [0.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930691)                    | Apple iPhone 15 Pro    |
 | MobileNetV2 (mv2)     | XNNPACK Q8   | [14.397](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379)               | [10.796](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987379)                     | Samsung S22            |
 | MobileNetV2 (mv2)     | QNN FP16     | [136.862](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097)              | [0.673](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218987097)                      | Samsung S22            |
-| MobileNetV2 (mv2)     | COREML FP16  | [63.019](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397)               | [551.031](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269520397)                    | Apple iPhone 15 Plus   |
+| MobileNetV2 (mv2)     | COREML FP16  | [50.892](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594)               | [0.631](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078594)                    | Apple iPhone 15 Pro   |
 | InceptionV4 (ic4)     | XNNPACK Q8   | [87.617](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716)               | [117.937](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986716)                    | Samsung S22            |
 | InceptionV4 (ic4)     | QNN FP16     | [163.943](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381)              | [2.734](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986381)                      | Samsung S22            |
-| InceptionV4 (ic4)     | COREML FP16  | [134.402](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983)              | [1906.966](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518983)                   | Apple iPhone 15 Pro    |
+| InceptionV4 (ic4)     | COREML FP16  | [118.686](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140)              | [4.289](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930140)                   | Apple iPhone 15 Pro    |
 | InceptionV3 (ic3)     | XNNPACK Q8   | [60.708](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023)               | [98.390](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218986023)                     | Samsung S22            |
 | InceptionV3 (ic3)     | QNN FP16     | [134.732](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425)              | [1.351](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985425)                      | Samsung S22            |
-| InceptionV3 (ic3)     | COREML FP16  | [91.808](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511)               | [1500.712](https://github.com/pytorch/executorch/actions/runs/10892026450/job/30269518511)                   | Apple iPhone 15 Pro    |
+| InceptionV3 (ic3)     | COREML FP16  | [86.728](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753)               | [1.391](https://github.com/pytorch/executorch/actions/runs/11171117103/job/31056078753)                   | Apple iPhone 15 Pro    |
 | DeepLabV3 (dl3)       | XNNPACK Q8   | [90.616](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758)               | [666.219](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985758)                    | Samsung S22            |
 | DeepLabV3 (dl3)       | QNN FP16     | [182.207](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141)              | [9.759](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218985141)                      | Samsung S22            |
 | ResNet50 (resnet50)   | XNNPACK Q8   | [55.462](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102)               | [37.021](https://github.com/pytorch/executorch/actions/runs/10911725781/job/30285857102)                     | Apple iPhone 15 Pro    |
-| ResNet50 (resnet50)   | COREML FP16  | [84.453](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731)               | [1329.846](https://github.com/pytorch/executorch/actions/runs/10836589365/job/30078680731)                   | Apple iPhone 15 Pro    |
+| ResNet50 (resnet50)   | COREML FP16  | [68.943](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818)               | [1.979](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930818)                   | Apple iPhone 15 Pro    |
 | TorchVisionVit (vit)  | QNN FP16     | [174.430](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581)              | [199.279](https://github.com/pytorch/executorch/actions/runs/10875550238/job/30218989581)                    | Samsung S22            |
 | Wave2Letter (w2l)     | XNNPACK Q8   | [33.913](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319)               | [135.584](https://github.com/pytorch/executorch/actions/runs/10857890364/job/30139445319)                    | Apple iPhone 15 Pro    |
+| Wave2Letter (w2l)     | COREML FP16  | [109.254](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566)               | [28.465](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999931566)                    | Apple iPhone 15    |
+| MobileBERT (mobilebert) | XNNPACK FP32 | [26.499](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558)               | [33.978](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930558)                    | Apple iPhone 15 Pro    |
+| MobileBERT (mobilebert) | COREML FP16  | [206.202](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398)               | [1.873](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999930398)                    | Apple iPhone 15 Pro    |
+| EDSR (edsr)           | XNNPACK Q8   | [3.190](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836)               | [168.429](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929836)                    | Apple iPhone 15 Pro    |
+| EDSR (edsr)           | COREML FP16  | [156.075](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690)               | [77.346](https://github.com/pytorch/executorch/actions/runs/11136241814/job/30999929690)                    | Apple iPhone 15 Pro    |
 
 
 ## Supported Use Cases