Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hoist benchmarking steps into reusable Github action #110

Merged
merged 1 commit into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions .github/actions/bench/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# SPDX-License-Identifier: Apache-2.0

name: Bench MLKEM-C-AArch64
description: Run benchmarking script

inputs:
name:
description: Name for the benchmarking run
required: true
perf:
description: Method of obtaining PMU metrics
required: true
default: 'PERF'
type: choice
options:
- NO
- PERF
- PMU
- M1
cflags:
description: CFLAGS to pass to compilation
default: ''
archflags:
description: ARCHFLAGS to pass to compilation
default: ''
bench_extra_args:
description: Further arguments to be appended to command line for `bench` script
default: ''
store_results:
description: Whether to push results to GH pages
default: 'false'
gh_token:
description: GitHub access token
required: true
runs:
using: composite
steps:
- name: Run benchmark
shell: nix develop .#ci -c bash -e {0}
run: |
tests bench -c ${{ inputs.perf }} --cflags "${{ inputs.cflags }}" --arch-flags "${{ inputs.archflags }}" -v --output output.json ${{ inputs.bench_extra_args }}
- name: Dump benchmark
shell: bash
if: ${{ inputs.store_results != 'true' }}
run: |
cat output.json
- name: Store benchmark result
if: ${{ inputs.store_results == 'true' }}
uses: benchmark-action/github-action-benchmark@v1
with:
name: ${{ inputs.name }}
tool: 'customSmallerIsBetter'
output-file-path: output.json
github-token: ${{ inputs.gh_token }}
auto-push: true
31 changes: 19 additions & 12 deletions .github/workflows/bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,22 @@ jobs:
target:
- system: rpi4
name: Arm Cortex-A72 (Raspberry Pi 4) benchmarks
cmd: tests bench -c PMU --arch-flags -mcpu=cortex-a72 -v --output output.json
bench_pmu: PMU
archflags: -mcpu=cortex-a72
cflags:
bench_extra_args:
- system: rpi5
name: Arm Cortex-A76 (Raspberry Pi 5) benchmarks
cmd: tests bench -c PERF --arch-flags "-mcpu=cortex-a76 -march=armv8.2-a" -v --output output.json
bench_pmu: PERF
archflags: "-mcpu=cortex-a76 -march=armv8.2-a"
cflags:
bench_extra_args:
potsrevennil marked this conversation as resolved.
Show resolved Hide resolved
- system: a55
name: Arm Cortex-A55 (Snapdragon 888) benchmarks
cmd: tests bench -c PERF --cflags -static --arch-flags "-mcpu=cortex-a55 -march=armv8.2-a" -w exec-on-a55 -v --output output.json
bench_pmu: PERF
archflags: "-mcpu=cortex-a55 -march=armv8.2-a"
cflags: -static
bench_extra_args: -w exec-on-a55
runs-on: self-hosted-${{ matrix.target.system }}
defaults:
run:
Expand Down Expand Up @@ -51,14 +60,12 @@ jobs:
$(cat /proc/cpuinfo)
EOF
- name: Run benchmark
run: |
${{ matrix.target.cmd }}
- name: Store benchmark result
if: github.repository_owner == 'pq-code-package' && github.ref == 'refs/heads/main'
uses: benchmark-action/github-action-benchmark@v1
uses: ./.github/actions/bench
with:
name: ${{ matrix.target.name }}
tool: 'customSmallerIsBetter'
output-file-path: output.json
github-token: ${{ secrets.GITHUB_TOKEN }}
auto-push: true
cflags: ${{ matrix.target.cflags }}
archflags: ${{ matrix.target.archflags }}
perf: ${{ matrix.target.bench_pmu }}
store_results: ${{ github.repository_owner == 'pq-code-package' && github.ref == 'refs/heads/main' }}
bench_extra_args: ${{ matrix.target.bench_extra_args }}
gh_token: ${{ secrets.GITHUB_TOKEN }}
4 changes: 4 additions & 0 deletions .github/workflows/bench_ec2_any.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,9 @@ on:
always_terminate:
description: Indicates if EC2 instance should always be terminated
default: true
bench_extra_args:
description: Additional command line to be appended to `tests bench` script
default: ''
jobs:
bench-ec2-any:
name: Ad-hoc benchmark on $${{ github.event.inputs.ec2_instance_type }}
Expand All @@ -35,4 +38,5 @@ jobs:
name: ${{ github.event.inputs.name }}
store_results: ${{ github.event.inputs.store_results }}
always_terminate: ${{ github.event.inputs.always_terminate }}
bench_extra_args: ${{ github.event.inputs.bench_extra_args }}
secrets: inherit
22 changes: 11 additions & 11 deletions .github/workflows/bench_ec2_reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ on:
type: string
description: Indicates if instance should always be terminated, even on failure
default: 'true'
bench_extra_args:
type: string
description: Additional command line to be appended to `bench` script
default: ''
env:
AWS_ROLE: arn:aws:iam::559050233797:role/mlkem-c-aarch64-gh-action
AWS_REGION: us-east-1
Expand Down Expand Up @@ -91,19 +95,15 @@ jobs:
$(cat /proc/cpuinfo)
EOF
- name: Run benchmark
shell: nix develop .#ci -c bash -e {0}
run: |
tests bench -c PERF --cflags "${{ inputs.cflags }}" --arch-flags "${{ inputs.archflags }}" -v --output output.json
- name: Store benchmark result
if: ${{ inputs.store_results == 'true' }}
uses: benchmark-action/github-action-benchmark@v1
uses: ./.github/actions/bench
with:
name: ${{ inputs.name }}
tool: 'customSmallerIsBetter'
output-file-path: output.json
github-token: ${{ secrets.AWS_GITHUB_TOKEN }}
auto-push: true

cflags: ${{ inputs.cflags }}
archflags: ${{ inputs.archflags }}
perf: PERF
store_results: ${{ inputs.store_results }}
bench_extra_args: ${{ inputs.bench_extra_args }}
gh_token: ${{ secrets.AWS_GITHUB_TOKEN }}
stop-ec2-runner:
name: Stop ${{ github.event.inputs.name }} (${{ github.event.inputs.ec2_instance_type }})
permissions:
Expand Down
Loading