Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move benchmark apps to extension/benchmark dir #5951

Closed
wants to merge 1 commit into from
Closed

Move benchmark apps to extension/benchmark dir #5951

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/android-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ jobs:
--output-dir benchmark-results \
--repo ${{ github.repository }} \
--head-branch ${{ github.head_ref || github.ref_name }} \
--workflow-name ${{ github.workflow }} \
--workflow-name "${{ github.workflow }}" \
--workflow-run-id ${{ github.run_id }} \
--workflow-run-attempt ${{ github.run_attempt }}
done
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/android.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ on:
- install_requirements.sh
- examples/demo-apps/android/**
- extension/android/**
- extension/benchmark/android/**
- extension/module/**
workflow_dispatch:

Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/apple-perf.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,17 +235,17 @@ jobs:
PYTHON_EXECUTABLE=python ${CONDA_RUN} --no-capture-output \
build/build_apple_frameworks.sh --coreml --custom --mps --optimized --portable --quantized --xnnpack
mkdir -p extension/apple/Benchmark/Frameworks
mkdir -p extension/benchmark/apple/Benchmark/Frameworks
for FRAMEWORK in "${FRAMEWORKS[@]}"; do (
cp -r "cmake-out/${FRAMEWORK}.xcframework" extension/apple/Benchmark/Frameworks/
cp -r "cmake-out/${FRAMEWORK}.xcframework" extension/benchmark/apple/Benchmark/Frameworks/
) done
echo "::endgroup::"
# NB: Although exported models can be copied to this directory and bundled together with the
# app, we don't use this in CI and rely on AWS extra data parameter to make the model and the
# tokenizer available to the benchmark. This decouples the app and the model. We just need to
# create the directory here to pass the build
mkdir -p extension/apple/Benchmark/Models
mkdir -p extension/benchmark/apple/Benchmark/Models
${CONDA_RUN} --no-capture-output \
build/build_apple_llm_demo.sh ${ARTIFACTS_DIR_NAME}
Expand Down
7 changes: 4 additions & 3 deletions .github/workflows/apple.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ on:
- build/test_ios_ci.sh
- examples/demo-apps/apple_ios/**
- extension/apple/**
- extension/benchmark/apple/**
- extension/module/**
workflow_dispatch:

Expand Down Expand Up @@ -272,14 +273,14 @@ jobs:
PYTHON_EXECUTABLE=python ${CONDA_RUN} --no-capture-output \
build/build_apple_frameworks.sh --coreml --custom --mps --optimized --portable --quantized --xnnpack
mkdir -p extension/apple/Benchmark/Frameworks
mkdir -p extension/benchmark/apple/Benchmark/Frameworks
for FRAMEWORK in "${FRAMEWORKS[@]}"; do (
cp -r "cmake-out/${FRAMEWORK}.xcframework" extension/apple/Benchmark/Frameworks/
cp -r "cmake-out/${FRAMEWORK}.xcframework" extension/benchmark/apple/Benchmark/Frameworks/
) done
echo "::endgroup::"
echo "::group::Build ExecuTorch benchmark app"
mkdir -p extension/apple/Benchmark/Models
mkdir -p extension/benchmark/apple/Benchmark/Models
${CONDA_RUN} --no-capture-output \
build/build_apple_llm_demo.sh ${ARTIFACTS_DIR_NAME}
echo "::endgroup::"
2 changes: 1 addition & 1 deletion .github/workflows/lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
FILES_NEEDS_FORMAT=$(/opt/google-java-format -n extension/android/src/main/java/org/pytorch/executorch/*.java \
examples/demo-apps/android/ExecuTorchDemo/app/src/main/java/com/example/executorchdemo/*.java \
examples/demo-apps/android/LlamaDemo/app/src/main/java/com/example/executorchllamademo/*.java \
extension/android/benchmark/app/src/main/java/org/pytorch/minibench/*.java)
extension/benchmark/android/benchmark/app/src/main/java/org/pytorch/minibench/*.java)
if [ -n "$FILES_NEEDS_FORMAT" ]; then
echo "Warning: The following files need formatting. Please use google-java-format."
echo "Use a binary from https://github.com/google/google-java-format/releases/"
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/upload-android-test-specs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ on:
pull_request:
paths:
- .github/workflows/upload-android-test-specs.yml
- extension/android/benchmark/android-llm-device-farm-test-spec.yml
- extension/benchmark/android/benchmark/android-llm-device-farm-test-spec.yml
push:
branches:
- main
paths:
- .github/workflows/upload-android-test-specs.yml
- extension/android/benchmark/android-llm-device-farm-test-spec.yml
- extension/benchmark/android/benchmark/android-llm-device-farm-test-spec.yml

concurrency:
# NB: This concurency group needs to be different than the one used in android-perf, otherwise
Expand All @@ -32,7 +32,7 @@ jobs:
${{ github.repository }}/${{ github.run_id }}/artifacts
retention-days: 1
if-no-files-found: error
path: extension/android/benchmark/android-llm-device-farm-test-spec.yml
path: extension/benchmark/android/benchmark/android-llm-device-farm-test-spec.yml

validate-android-test-spec:
needs: upload-android-test-spec-for-validation
Expand Down Expand Up @@ -77,7 +77,7 @@ jobs:
- name: Upload the spec to S3 ossci-android bucket
shell: bash
working-directory: extension/android/benchmark/
working-directory: extension/benchmark/android/benchmark/
env:
SPEC_FILE: android-llm-device-farm-test-spec.yml
run: |
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/upload-apple-test-specs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ on:
pull_request:
paths:
- .github/workflows/upload-apple-test-specs.yml
- examples/demo-apps/apple_ios/default-ios-device-farm-appium-test-spec.yml
- extension/benchmark/apple/Benchmark/default-ios-device-farm-appium-test-spec.yml
push:
branches:
- main
paths:
- .github/workflows/upload-apple-test-specs.yml
- examples/demo-apps/apple_ios/default-ios-device-farm-appium-test-spec.yml
- extension/benchmark/apple/Benchmark/default-ios-device-farm-appium-test-spec.yml

concurrency:
# NB: This concurency group needs to be different than the one used in apple-perf, otherwise
Expand All @@ -32,7 +32,7 @@ jobs:
${{ github.repository }}/${{ github.run_id }}/artifacts
retention-days: 1
if-no-files-found: error
path: examples/demo-apps/apple_ios/default-ios-device-farm-appium-test-spec.yml
path: extension/benchmark/apple/Benchmark/default-ios-device-farm-appium-test-spec.yml

validate-apple-test-spec:
needs: upload-apple-test-spec-for-validation
Expand Down Expand Up @@ -78,7 +78,7 @@ jobs:
- name: Upload the spec to S3 ossci-ios bucket
shell: bash
working-directory: examples/demo-apps/apple_ios
working-directory: extension/benchmark/apple/Benchmark/
env:
SPEC_FILE: default-ios-device-farm-appium-test-spec.yml
run: |
Expand Down
10 changes: 5 additions & 5 deletions build/build_android_llm_demo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,9 @@ build_android_demo_apps() {
ANDROID_HOME="${ANDROID_SDK:-/opt/android/sdk}" ./gradlew build assembleAndroidTest
popd

mkdir -p extension/android/benchmark/app/libs
cp ${BUILD_AAR_DIR}/executorch.aar extension/android/benchmark/app/libs
pushd extension/android/benchmark
mkdir -p extension/benchmark/android/benchmark/app/libs
cp ${BUILD_AAR_DIR}/executorch.aar extension/benchmark/android/benchmark/app/libs
pushd extension/benchmark/android/benchmark
ANDROID_HOME="${ANDROID_SDK:-/opt/android/sdk}" ./gradlew build assembleAndroidTest
popd
}
Expand All @@ -135,8 +135,8 @@ collect_artifacts_to_be_uploaded() {
# Collect MiniBench APK
MINIBENCH_APP_DIR="${ARTIFACTS_DIR_NAME}/minibench"
mkdir -p "${MINIBENCH_APP_DIR}"
cp extension/android/benchmark/app/build/outputs/apk/debug/*.apk "${MINIBENCH_APP_DIR}"
cp extension/android/benchmark/app/build/outputs/apk/androidTest/debug/*.apk "${MINIBENCH_APP_DIR}"
cp extension/benchmark/android/benchmark/app/build/outputs/apk/debug/*.apk "${MINIBENCH_APP_DIR}"
cp extension/benchmark/android/benchmark/app/build/outputs/apk/androidTest/debug/*.apk "${MINIBENCH_APP_DIR}"
}

BUILD_AAR_DIR="$(mktemp -d)"
Expand Down
2 changes: 1 addition & 1 deletion build/build_apple_llm_demo.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
set -euo pipefail

ARTIFACTS_DIR_NAME="$1"
APP_PATH="extension/apple/Benchmark/Benchmark"
APP_PATH="extension/benchmark/apple/Benchmark/Benchmark"

xcodebuild build-for-testing \
-project "${APP_PATH}.xcodeproj" \
Expand Down
0 extension/android/benchmark/.gitignore → ...on/benchmark/android/benchmark/.gitignore
View file
Open in desktop
File renamed without changes.
0 extension/android/benchmark/README.md → ...ion/benchmark/android/benchmark/README.md
View file
Open in desktop
File renamed without changes.
0 ...ark/android-llm-device-farm-test-spec.yml → ...ark/android-llm-device-farm-test-spec.yml
View file
Open in desktop
File renamed without changes.
0 extension/android/benchmark/app/.gitignore → ...enchmark/android/benchmark/app/.gitignore
View file
Open in desktop
File renamed without changes.
0 ...on/android/benchmark/app/build.gradle.kts → ...rk/android/benchmark/app/build.gradle.kts
View file
Open in desktop
File renamed without changes.
0 .../android/benchmark/app/proguard-rules.pro → .../android/benchmark/app/proguard-rules.pro
View file
Open in desktop
File renamed without changes.
0 ...ch/minibench/ExampleInstrumentedTest.java → ...ch/minibench/ExampleInstrumentedTest.java
View file
Open in desktop
File renamed without changes.
0 ...enchmark/app/src/main/AndroidManifest.xml → ...enchmark/app/src/main/AndroidManifest.xml
View file
Open in desktop
File renamed without changes.
0 .../pytorch/minibench/BenchmarkActivity.java → .../pytorch/minibench/BenchmarkActivity.java
View file
Open in desktop
File renamed without changes.
0 ...rg/pytorch/minibench/BenchmarkMetric.java → ...rg/pytorch/minibench/BenchmarkMetric.java
View file
Open in desktop
File renamed without changes.
0 ...torch/minibench/LlmBenchmarkActivity.java → ...torch/minibench/LlmBenchmarkActivity.java
View file
Open in desktop
File renamed without changes.
0 ...va/org/pytorch/minibench/ModelRunner.java → ...va/org/pytorch/minibench/ModelRunner.java
View file
Open in desktop
File renamed without changes.
0 ...ytorch/minibench/ModelRunnerCallback.java → ...ytorch/minibench/ModelRunnerCallback.java
View file
Open in desktop
File renamed without changes.
0 ...chmark/app/src/main/res/values/colors.xml → ...chmark/app/src/main/res/values/colors.xml
View file
Open in desktop
File renamed without changes.
0 ...hmark/app/src/main/res/values/strings.xml → ...hmark/app/src/main/res/values/strings.xml
View file
Open in desktop
File renamed without changes.
0 ...chmark/app/src/main/res/values/themes.xml → ...chmark/app/src/main/res/values/themes.xml
View file
Open in desktop
File renamed without changes.
0 ...rg/pytorch/minibench/ExampleUnitTest.java → ...rg/pytorch/minibench/ExampleUnitTest.java
View file
Open in desktop
File renamed without changes.
0 extension/android/benchmark/build.gradle.kts → ...chmark/android/benchmark/build.gradle.kts
View file
Open in desktop
File renamed without changes.
0 ...nsion/android/benchmark/gradle.properties → ...hmark/android/benchmark/gradle.properties
View file
Open in desktop
File renamed without changes.
0 ...nchmark/gradle/wrapper/gradle-wrapper.jar → ...nchmark/gradle/wrapper/gradle-wrapper.jar
View file
Open in desktop
File renamed without changes.
0 .../gradle/wrapper/gradle-wrapper.properties → .../gradle/wrapper/gradle-wrapper.properties
View file
Open in desktop
File renamed without changes.
0 extension/android/benchmark/gradlew → ...nsion/benchmark/android/benchmark/gradlew
View file
Open in desktop
File renamed without changes.
0 extension/android/benchmark/gradlew.bat → ...n/benchmark/android/benchmark/gradlew.bat
View file
Open in desktop
File renamed without changes.
0 ...ion/android/benchmark/settings.gradle.kts → ...ark/android/benchmark/settings.gradle.kts
View file
Open in desktop
File renamed without changes.
45 changes: 45 additions & 0 deletions extension/benchmark/apple/.clang-format
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
---
AccessModifierOffset: -1
AlignEscapedNewlinesLeft: true
AlignTrailingComments: false
AllowAllParametersOfDeclarationOnNextLine: true
AllowShortIfStatementsOnASingleLine: false
AllowShortLoopsOnASingleLine: false
AlwaysBreakBeforeMultilineStrings: true
AlwaysBreakTemplateDeclarations: true
BinPackArguments: false
BinPackParameters: false
BreakBeforeBinaryOperators: false
BreakBeforeBraces: Attach
BreakConstructorInitializersBeforeComma: false
ColumnLimit: 80
ConstructorInitializerAllOnOneLineOrOnePerLine: true
ConstructorInitializerIndentWidth: 4
ContinuationIndentWidth: 4
Cpp11BracedListStyle: true
DerivePointerAlignment: false
ExperimentalAutoDetectBinPacking: true
IndentCaseLabels: false
IndentFunctionDeclarationAfterType: false
IndentWidth: 2
MaxEmptyLinesToKeep: 1
NamespaceIndentation: None
ObjCSpaceBeforeProtocolList: false
PenaltyBreakBeforeFirstCallParameter: 10
PenaltyBreakComment: 60
PenaltyBreakFirstLessLess: 20
PenaltyBreakString: 1000
PenaltyExcessCharacter: 1000000
PenaltyReturnTypeOnItsOwnLine: 200
PointerAlignment: Right
SpaceAfterControlStatementKeyword: true
SpaceBeforeAssignmentOperators: true
SpaceInEmptyParentheses: false
SpacesBeforeTrailingComments: 1
SpacesInAngles: false
SpacesInCStyleCastParentheses: false
SpacesInParentheses: false
Standard: Cpp11
TabWidth: 8
UseTab: Never
...
0 ...sion/apple/Benchmark/App/App.entitlements → ...mark/apple/Benchmark/App/App.entitlements
View file
Open in desktop
File renamed without changes.
0 extension/apple/Benchmark/App/App.swift → ...n/benchmark/apple/Benchmark/App/App.swift
View file
Open in desktop
File renamed without changes.
16 changes: 8 additions & 8 deletions ...hmark/Benchmark.xcodeproj/project.pbxproj → ...hmark/Benchmark.xcodeproj/project.pbxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,10 @@
032A73FD2CAFBB7800932D36 /* tiktoken.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = tiktoken.h; sourceTree = "<group>"; };
032A73FE2CAFBB7800932D36 /* tiktoken.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; path = tiktoken.cpp; sourceTree = "<group>"; };
032A73FF2CAFBB7800932D36 /* tokenizer.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = tokenizer.h; sourceTree = "<group>"; };
032A74212CAFC1B300932D36 /* runner.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = runner.h; path = ../../../examples/models/llama2/runner/runner.h; sourceTree = SOURCE_ROOT; };
032A74222CAFC1B300932D36 /* runner.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; name = runner.cpp; path = ../../../examples/models/llama2/runner/runner.cpp; sourceTree = SOURCE_ROOT; };
032A74242CAFC34800932D36 /* llama_tiktoken.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = llama_tiktoken.h; path = ../../../examples/models/llama2/tokenizer/llama_tiktoken.h; sourceTree = SOURCE_ROOT; };
032A74252CAFC34800932D36 /* llama_tiktoken.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; name = llama_tiktoken.cpp; path = ../../../examples/models/llama2/tokenizer/llama_tiktoken.cpp; sourceTree = SOURCE_ROOT; };
032A74212CAFC1B300932D36 /* runner.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = runner.h; path = ../../../../examples/models/llama2/runner/runner.h; sourceTree = SOURCE_ROOT; };
032A74222CAFC1B300932D36 /* runner.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; name = runner.cpp; path = ../../../../examples/models/llama2/runner/runner.cpp; sourceTree = SOURCE_ROOT; };
032A74242CAFC34800932D36 /* llama_tiktoken.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; name = llama_tiktoken.h; path = ../../../../examples/models/llama2/tokenizer/llama_tiktoken.h; sourceTree = SOURCE_ROOT; };
032A74252CAFC34800932D36 /* llama_tiktoken.cpp */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.cpp.cpp; name = llama_tiktoken.cpp; path = ../../../../examples/models/llama2/tokenizer/llama_tiktoken.cpp; sourceTree = SOURCE_ROOT; };
037C96A02C8A570B00B3DF38 /* Tests.xctestplan */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text; path = Tests.xctestplan; sourceTree = "<group>"; };
03B0118B2CAC567900054791 /* DynamicTestCase.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = DynamicTestCase.h; sourceTree = "<group>"; };
03B0118C2CAC567900054791 /* DynamicTestCase.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = DynamicTestCase.m; sourceTree = "<group>"; };
Expand Down Expand Up @@ -158,7 +158,7 @@
032A73DF2CAFBB7800932D36 /* util.h */,
);
name = runner;
path = ../../llm/runner;
path = ../../../llm/runner;
sourceTree = SOURCE_ROOT;
};
032A73E92CAFBB7800932D36 /* sampler */ = {
Expand All @@ -168,7 +168,7 @@
032A73E62CAFBB7800932D36 /* sampler.cpp */,
);
name = sampler;
path = ../../llm/sampler;
path = ../../../llm/sampler;
sourceTree = SOURCE_ROOT;
};
032A74022CAFBB7800932D36 /* tokenizer */ = {
Expand All @@ -184,7 +184,7 @@
032A73FF2CAFBB7800932D36 /* tokenizer.h */,
);
name = tokenizer;
path = ../../llm/tokenizer;
path = ../../../llm/tokenizer;
sourceTree = SOURCE_ROOT;
};
03B0118D2CAC567900054791 /* TestUtils */ = {
Expand Down Expand Up @@ -364,7 +364,7 @@
);
runOnlyForDeploymentPostprocessing = 0;
shellPath = /bin/sh;
shellScript = "set -e\n\nif ! command -v cmake &> /dev/null\nthen\n echo \"Cmake not found, please install Cmake. \\n1. Download Cmake.app from https://cmake.org/download with version > 3.19. \\n2. Install it to Applications/ folder and run sudo /Applications/CMake.app/Contents/bin/cmake-gui --install to install CMake commandline tools.\"\n exit 1\nfi\n\nCMAKE_DIR=\"$TEMP_DIR/cmake\"\nrm -rf \"$CMAKE_DIR\"\n\nPLATFORM=\"SIMULATORARM64\"\nDEPLOYMENT_TARGET=\"17.0\"\n\nif [[ \"$PLATFORM_NAME\" == *\"iphoneos\"* ]]; then\n PLATFORM=\"OS64\"\nelif [[ \"$PLATFORM_NAME\" == *\"macos\"* ]]; then\n PLATFORM=\"MAC_ARM64\"\n DEPLOYMENT_TARGET=\"10.15\"\nfi\n\ncmake_build() {\n local src_dir=$1\n local target=$2\n shift 2\n local extra_args=(\"$@\")\n local build_dir=\"$CMAKE_DIR/build/$(basename \"$src_dir\")\"\n\n mkdir -p \"$build_dir\" && cd \"$build_dir\"\n\n if [[ \"$PLATFORM\" == \"MAC_ARM64\" ]]; then\n extra_args+=(-DCMAKE_INSTALL_BUNDLEDIR=\"${CMAKE_DIR}/bin\")\n extra_args+=(-DCMAKE_MACOSX_BUNDLE=OFF)\n fi\n cmake -G Xcode \\\n -DCMAKE_BUILD_TYPE=\"Release\" \\\n -DCMAKE_CXX_STANDARD=17 \\\n -DCMAKE_TOOLCHAIN_FILE=\"$SRCROOT/../../../third-party/ios-cmake/ios.toolchain.cmake\" \\\n -DCMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LANGUAGE_STANDARD=\"c++17\" \\\n -DCMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LIBRARY=\"libc++\" \\\n -DPLATFORM=\"$PLATFORM\" \\\n -DDEPLOYMENT_TARGET=\"$DEPLOYMENT_TARGET\" \\\n -DCMAKE_INSTALL_PREFIX=\"$CMAKE_DIR\" \\\n \"${extra_args[@]}\" \\\n \"$src_dir\"\n cmake --build . --config \"Release\" --target \"$target\"\n if [[ \"$target\" == \"install\" ]]; then\n cmake --install . --prefix \"$CMAKE_DIR\"\n fi\n}\n\ncmake_build \"$SRCROOT/../../llm/third-party/abseil-cpp\" \"install\" \\\n -DABSL_PROPAGATE_CXX_STD=ON\n\ncmake_build \"$SRCROOT/../../llm/third-party/re2\" \"install\"\n\ncmake_build \"$SRCROOT/../../llm/third-party/sentencepiece\" \"sentencepiece-static\" \\\n -DSPM_ENABLE_SHARED=OFF\n\necho \"$(find $CMAKE_DIR/lib -name \"*.a\" | sed -E 's|^.*/lib([^/]+)\\.a|-l\\1|g' | tr '\\n' ' ')\" > \"$CMAKE_DIR/linker_flags\"\n";
shellScript = "set -e\n\nif ! command -v cmake &> /dev/null\nthen\n echo \"Cmake not found, please install Cmake. \\n1. Download Cmake.app from https://cmake.org/download with version > 3.19. \\n2. Install it to Applications/ folder and run sudo /Applications/CMake.app/Contents/bin/cmake-gui --install to install CMake commandline tools.\"\n exit 1\nfi\n\nCMAKE_DIR=\"$TEMP_DIR/cmake\"\nrm -rf \"$CMAKE_DIR\"\n\nPLATFORM=\"SIMULATORARM64\"\nDEPLOYMENT_TARGET=\"17.0\"\n\nif [[ \"$PLATFORM_NAME\" == *\"iphoneos\"* ]]; then\n PLATFORM=\"OS64\"\nelif [[ \"$PLATFORM_NAME\" == *\"macos\"* ]]; then\n PLATFORM=\"MAC_ARM64\"\n DEPLOYMENT_TARGET=\"10.15\"\nfi\n\ncmake_build() {\n local src_dir=$1\n local target=$2\n shift 2\n local extra_args=(\"$@\")\n local build_dir=\"$CMAKE_DIR/build/$(basename \"$src_dir\")\"\n\n mkdir -p \"$build_dir\" && cd \"$build_dir\"\n\n if [[ \"$PLATFORM\" == \"MAC_ARM64\" ]]; then\n extra_args+=(-DCMAKE_INSTALL_BUNDLEDIR=\"${CMAKE_DIR}/bin\")\n extra_args+=(-DCMAKE_MACOSX_BUNDLE=OFF)\n fi\n cmake -G Xcode \\\n -DCMAKE_BUILD_TYPE=\"Release\" \\\n -DCMAKE_CXX_STANDARD=17 \\\n -DCMAKE_TOOLCHAIN_FILE=\"$SRCROOT/../../../../third-party/ios-cmake/ios.toolchain.cmake\" \\\n -DCMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LANGUAGE_STANDARD=\"c++17\" \\\n -DCMAKE_XCODE_ATTRIBUTE_CLANG_CXX_LIBRARY=\"libc++\" \\\n -DPLATFORM=\"$PLATFORM\" \\\n -DDEPLOYMENT_TARGET=\"$DEPLOYMENT_TARGET\" \\\n -DCMAKE_INSTALL_PREFIX=\"$CMAKE_DIR\" \\\n \"${extra_args[@]}\" \\\n \"$src_dir\"\n cmake --build . --config \"Release\" --target \"$target\"\n if [[ \"$target\" == \"install\" ]]; then\n cmake --install . --prefix \"$CMAKE_DIR\"\n fi\n}\n\ncmake_build \"$SRCROOT/../../../llm/third-party/abseil-cpp\" \"install\" \\\n -DABSL_PROPAGATE_CXX_STD=ON\n\ncmake_build \"$SRCROOT/../../../llm/third-party/re2\" \"install\"\n\ncmake_build \"$SRCROOT/../../../llm/third-party/sentencepiece\" \"sentencepiece-static\" \\\n -DSPM_ENABLE_SHARED=OFF\n\necho \"$(find $CMAKE_DIR/lib -name \"*.a\" | sed -E 's|^.*/lib([^/]+)\\.a|-l\\1|g' | tr '\\n' ' ')\" > \"$CMAKE_DIR/linker_flags\"\n";
};
/* End PBXShellScriptBuildPhase section */

Expand Down
0 ...xcshareddata/xcschemes/Benchmark.xcscheme → ...xcshareddata/xcschemes/Benchmark.xcscheme
View file
Open in desktop
File renamed without changes.
0 ...nchmark/Frameworks/download_frameworks.sh → ...nchmark/Frameworks/download_frameworks.sh
View file
Open in desktop
File renamed without changes.
16 changes: 8 additions & 8 deletions extension/apple/Benchmark/README.md → ...nsion/benchmark/apple/Benchmark/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ You have two options:
Run the provided script to download the prebuilt frameworks:

```bash
./extension/apple/Benchmark/Frameworks/download_frameworks.sh
./extension/benchmark/apple/Benchmark/Frameworks/download_frameworks.sh
```
</details>

Expand All @@ -56,7 +56,7 @@ Alternatively, you can build the frameworks yourself by following the [guide](ht
Once the frameworks are downloaded or built, verify that the `Frameworks` directory contains the necessary `.xcframework` files:

```bash
ls extension/apple/Benchmark/Frameworks
ls extension/benchmark/apple/Benchmark/Frameworks
```

You should see:
Expand All @@ -74,16 +74,16 @@ kernels_quantized.xcframework

## Adding Models and Resources

Place your exported model files (`.pte`) and any other resources (e.g., `tokenizer.bin`) into the `extension/apple/Benchmark/Resources` directory:
Place your exported model files (`.pte`) and any other resources (e.g., `tokenizer.bin`) into the `extension/benchmark/apple/Benchmark/Resources` directory:

```bash
cp <path/to/my_model.pte> <path/to/llama3.pte> <path/to/tokenizer.bin> extension/apple/Benchmark/Resources
cp <path/to/my_model.pte> <path/to/llama3.pte> <path/to/tokenizer.bin> extension/benchmark/apple/Benchmark/Resources
```

Optionally, check that the files are there:

```bash
ls extension/apple/Benchmark/Resources
ls extension/benchmark/apple/Benchmark/Resources
```

For this example you should see:
Expand All @@ -103,7 +103,7 @@ The app automatically bundles these resources and makes them available to the te
Open the Benchmark Xcode project:

```bash
open extension/apple/Benchmark/Benchmark.xcodeproj
open extension/benchmark/apple/Benchmark/Benchmark.xcodeproj
```

Select the destination device or simulator and press `Command+U`, or click `Product` > `Test` in the menu to run the test suite.
Expand Down Expand Up @@ -333,13 +333,13 @@ You can also run the tests using `xcodebuild`:
```bash
# Run on an iOS Simulator
xcodebuild test -project extension/apple/Benchmark/Benchmark.xcodeproj \
xcodebuild test -project extension/benchmark/apple/Benchmark/Benchmark.xcodeproj \
-scheme Benchmark \
-destination 'platform=iOS Simulator,name=<SimulatorName>' \
-testPlan Tests
# Run on a physical iOS device
xcodebuild test -project extension/apple/Benchmark/Benchmark.xcodeproj \
xcodebuild test -project extension/benchmark/apple/Benchmark/Benchmark.xcodeproj \
-scheme Benchmark \
-destination 'platform=iOS,name=<DeviceName>' \
-testPlan Tests \
Expand Down
0 ...nsion/apple/Benchmark/Resources/README.md → ...hmark/apple/Benchmark/Resources/README.md
View file
Open in desktop
File renamed without changes.
0 ...ple/Benchmark/TestUtils/DynamicTestCase.h → ...ple/Benchmark/TestUtils/DynamicTestCase.h
View file
Open in desktop
File renamed without changes.
0 ...ple/Benchmark/TestUtils/DynamicTestCase.m → ...ple/Benchmark/TestUtils/DynamicTestCase.m
View file
Open in desktop
File renamed without changes.
0 ...le/Benchmark/TestUtils/ResourceTestCase.h → ...le/Benchmark/TestUtils/ResourceTestCase.h
View file
Open in desktop
File renamed without changes.
0 ...le/Benchmark/TestUtils/ResourceTestCase.m → ...le/Benchmark/TestUtils/ResourceTestCase.m
View file
Open in desktop
File renamed without changes.
0 ...ion/apple/Benchmark/Tests/GenericTests.mm → ...ark/apple/Benchmark/Tests/GenericTests.mm
View file
Open in desktop
File renamed without changes.
0 ...apple/Benchmark/Tests/LLaMA/LLaMATests.mm → ...apple/Benchmark/Tests/LLaMA/LLaMATests.mm
View file
Open in desktop
File renamed without changes.
Loading
Loading