Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarking for compute units usage #2466

Merged
merged 15 commits into from
Apr 24, 2023
Merged

Add benchmarking for compute units usage #2466

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
d9545a0
Add benchmark tests
acheroncrypto Apr 16, 2023
9ecf454
Fix last version Markdown table
acheroncrypto Apr 16, 2023
55f5adb
Trigger CI in bench branch
acheroncrypto Apr 16, 2023
2990e95
Downgrade node version for CI test
acheroncrypto Apr 16, 2023
5924f55
Error when a change has not been commited
acheroncrypto Apr 16, 2023
1b4d422
Reset to initial bench data and improve text
acheroncrypto Apr 16, 2023
db9dff8
Remove logging unreleased section when bumping version
acheroncrypto Apr 16, 2023
af18fba
Update version-bump script to include benchmark files
acheroncrypto Apr 17, 2023
376d492
Read bench dir instead of hardcoding file names
acheroncrypto Apr 18, 2023
079a06b
Generate all version results(Markdown) based on the data file on each…
acheroncrypto Apr 18, 2023
6409c25
Add a note in compute units markdown description
acheroncrypto Apr 23, 2023
ca552e6
Cleanup tests
acheroncrypto Apr 23, 2023
e3b4452
Re-enable other tests
acheroncrypto Apr 23, 2023
d2c313b
Merge master
acheroncrypto Apr 23, 2023
977b2b0
Update CHANGELOG
acheroncrypto Apr 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/reusable-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -437,6 +437,8 @@ jobs:
path: tests/anchor-cli-idl
- cmd: cd tests/anchor-cli-account && anchor test --skip-lint
path: tests/anchor-cli-account
- cmd: cd tests/bench && anchor test --skip-lint
path: tests/bench
steps:
- uses: actions/checkout@v3
- uses: ./.github/actions/setup/
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,15 @@ The minor version will be incremented upon a breaking change and the patch versi
- spl: Add metadata wrappers `approve_collection_authority`, `bubblegum_set_collection_size`, `burn_edition_nft`, `burn_nft`, `revoke_collection_authority`, `set_token_standard`, `utilize`, `unverify_sized_collection_item`, `unverify_collection` ([#2430](https://github.com/coral-xyz/anchor/pull/2430))
- spl: Add `token_program` constraint to `Token`, `Mint`, and `AssociatedToken` accounts in order to override required `token_program` fields and use different token interface implementations in the same instruction ([#2460](https://github.com/coral-xyz/anchor/pull/2460))
- cli: Add support for Solidity programs. `anchor init` and `anchor new` take an option `--solidity` which creates solidity code rather than rust. `anchor build` and `anchor test` work accordingly ([#2421](https://github.com/coral-xyz/anchor/pull/2421))
- bench: Add benchmarking for compute units usage ([#2466](https://github.com/coral-xyz/anchor/pull/2466))

### Fixes

- ts: Narrowed `AccountClient` type to it's appropriate account type ([#2440](https://github.com/coral-xyz/anchor/pull/2440))
- lang: Fix inability to use identifiers `program_id`, `accounts`, `ix_data`, `remaining_accounts` in instruction arguments ([#2464](https://github.com/coral-xyz/anchor/pull/2464))

### Breaking

- lang: Identifiers that are intended for internal usage(`program_id`, `accounts`, `ix_data`, `remaining_accounts`) have been renamed with `__` prefix ([#2464](https://github.com/coral-xyz/anchor/pull/2464))

## [0.27.0] - 2023-03-08
Expand Down
200 changes: 200 additions & 0 deletions bench/COMPUTE_UNITS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,200 @@
# Compute Units

All notable changes in compute units usage will be documented in this file.

The changes are calculated by comparing the current results with the last version's results. Increase in usage is shown with 🔴 and decrease is shown with 🟢.

The programs and their tests are located in [/tests/bench](https://github.com/coral-xyz/anchor/tree/master/tests/bench).

> **Note**
> The results documented in this file are autogenerated. Running the tests will update the current results when necessary, manually editing the results should be avoided.

## [Unreleased]

| Instruction | Compute Units | +/- |
| --------------------------- | ------------- | --- |
| accountInfo1 | 954 | - |
| accountInfo2 | 1567 | - |
| accountInfo4 | 2059 | - |
| accountInfo8 | 3856 | - |
| accountEmptyInit1 | 5958 | - |
| accountEmpty1 | 1090 | - |
| accountEmptyInit2 | 10583 | - |
| accountEmpty2 | 1852 | - |
| accountEmptyInit4 | 19557 | - |
| accountEmpty4 | 2646 | - |
| accountEmptyInit8 | 37541 | - |
| accountEmpty8 | 5043 | - |
| accountSizedInit1 | 6063 | - |
| accountSized1 | 1135 | - |
| accountSizedInit2 | 10783 | - |
| accountSized2 | 1966 | - |
| accountSizedInit4 | 19975 | - |
| accountSized4 | 2787 | - |
| accountSizedInit8 | 38381 | - |
| accountSized8 | 5359 | - |
| accountUnsizedInit1 | 6193 | - |
| accountUnsized1 | 1243 | - |
| accountUnsizedInit2 | 11042 | - |
| accountUnsized2 | 1893 | - |
| accountUnsizedInit4 | 20495 | - |
| accountUnsized4 | 3104 | - |
| accountUnsizedInit8 | 39419 | - |
| accountUnsized8 | 6051 | - |
| boxedAccountEmptyInit1 | 6160 | - |
| boxedAccountEmpty1 | 976 | - |
| boxedAccountEmptyInit2 | 10784 | - |
| boxedAccountEmpty2 | 1499 | - |
| boxedAccountEmptyInit4 | 19500 | - |
| boxedAccountEmpty4 | 2530 | - |
| boxedAccountEmptyInit8 | 37415 | - |
| boxedAccountEmpty8 | 4780 | - |
| boxedAccountSizedInit1 | 6256 | - |
| boxedAccountSized1 | 1003 | - |
| boxedAccountSizedInit2 | 10975 | - |
| boxedAccountSized2 | 1554 | - |
| boxedAccountSizedInit4 | 19884 | - |
| boxedAccountSized4 | 2642 | - |
| boxedAccountSizedInit8 | 38182 | - |
| boxedAccountSized8 | 5003 | - |
| boxedAccountUnsizedInit1 | 6374 | - |
| boxedAccountUnsized1 | 1069 | - |
| boxedAccountUnsizedInit2 | 11211 | - |
| boxedAccountUnsized2 | 1679 | - |
| boxedAccountUnsizedInit4 | 20351 | - |
| boxedAccountUnsized4 | 2899 | - |
| boxedAccountUnsizedInit8 | 39118 | - |
| boxedAccountUnsized8 | 5517 | - |
| boxedInterfaceAccountMint1 | 2299 | - |
| boxedInterfaceAccountMint2 | 4053 | - |
| boxedInterfaceAccountMint4 | 7538 | - |
| boxedInterfaceAccountMint8 | 14699 | - |
| boxedInterfaceAccountToken1 | 1737 | - |
| boxedInterfaceAccountToken2 | 2928 | - |
| boxedInterfaceAccountToken4 | 5291 | - |
| boxedInterfaceAccountToken8 | 10205 | - |
| interfaceAccountMint1 | 2530 | - |
| interfaceAccountMint2 | 4726 | - |
| interfaceAccountMint4 | 9431 | - |
| interfaceAccountMint8 | 17709 | - |
| interfaceAccountToken1 | 1755 | - |
| interfaceAccountToken2 | 3211 | - |
| interfaceAccountToken4 | 6006 | - |
| interface1 | 999 | - |
| interface2 | 1574 | - |
| interface4 | 1996 | - |
| interface8 | 3651 | - |
| program1 | 999 | - |
| program2 | 1573 | - |
| program4 | 1998 | - |
| program8 | 3651 | - |
| signer1 | 958 | - |
| signer2 | 1576 | - |
| signer4 | 2079 | - |
| signer8 | 3895 | - |
| systemAccount1 | 1013 | - |
| systemAccount2 | 1686 | - |
| systemAccount4 | 2298 | - |
| systemAccount8 | 4336 | - |
| uncheckedAccount1 | 953 | - |
| uncheckedAccount2 | 1567 | - |
| uncheckedAccount4 | 2060 | - |
| uncheckedAccount8 | 3855 | - |

### Notable changes

---

## [0.27.0]

| Instruction | Compute Units | +/- |
| --------------------------- | ------------- | --- |
| accountInfo1 | 954 | N/A |
| accountInfo2 | 1567 | N/A |
| accountInfo4 | 2059 | N/A |
| accountInfo8 | 3856 | N/A |
| accountEmptyInit1 | 5958 | N/A |
| accountEmpty1 | 1090 | N/A |
| accountEmptyInit2 | 10574 | N/A |
| accountEmpty2 | 1852 | N/A |
| accountEmptyInit4 | 19557 | N/A |
| accountEmpty4 | 2646 | N/A |
| accountEmptyInit8 | 37541 | N/A |
| accountEmpty8 | 5043 | N/A |
| accountSizedInit1 | 6063 | N/A |
| accountSized1 | 1135 | N/A |
| accountSizedInit2 | 10783 | N/A |
| accountSized2 | 1966 | N/A |
| accountSizedInit4 | 19975 | N/A |
| accountSized4 | 2787 | N/A |
| accountSizedInit8 | 38381 | N/A |
| accountSized8 | 5359 | N/A |
| accountUnsizedInit1 | 6193 | N/A |
| accountUnsized1 | 1243 | N/A |
| accountUnsizedInit2 | 11042 | N/A |
| accountUnsized2 | 1893 | N/A |
| accountUnsizedInit4 | 20495 | N/A |
| accountUnsized4 | 3104 | N/A |
| accountUnsizedInit8 | 39419 | N/A |
| accountUnsized8 | 6051 | N/A |
| boxedAccountEmptyInit1 | 6160 | N/A |
| boxedAccountEmpty1 | 976 | N/A |
| boxedAccountEmptyInit2 | 10784 | N/A |
| boxedAccountEmpty2 | 1499 | N/A |
| boxedAccountEmptyInit4 | 19500 | N/A |
| boxedAccountEmpty4 | 2530 | N/A |
| boxedAccountEmptyInit8 | 37415 | N/A |
| boxedAccountEmpty8 | 4780 | N/A |
| boxedAccountSizedInit1 | 6256 | N/A |
| boxedAccountSized1 | 1003 | N/A |
| boxedAccountSizedInit2 | 10975 | N/A |
| boxedAccountSized2 | 1554 | N/A |
| boxedAccountSizedInit4 | 19884 | N/A |
| boxedAccountSized4 | 2642 | N/A |
| boxedAccountSizedInit8 | 38182 | N/A |
| boxedAccountSized8 | 5003 | N/A |
| boxedAccountUnsizedInit1 | 6374 | N/A |
| boxedAccountUnsized1 | 1069 | N/A |
| boxedAccountUnsizedInit2 | 11211 | N/A |
| boxedAccountUnsized2 | 1679 | N/A |
| boxedAccountUnsizedInit4 | 20351 | N/A |
| boxedAccountUnsized4 | 2899 | N/A |
| boxedAccountUnsizedInit8 | 39118 | N/A |
| boxedAccountUnsized8 | 5517 | N/A |
| boxedInterfaceAccountMint1 | 2299 | N/A |
| boxedInterfaceAccountMint2 | 4053 | N/A |
| boxedInterfaceAccountMint4 | 7538 | N/A |
| boxedInterfaceAccountMint8 | 14699 | N/A |
| boxedInterfaceAccountToken1 | 1737 | N/A |
| boxedInterfaceAccountToken2 | 2928 | N/A |
| boxedInterfaceAccountToken4 | 5291 | N/A |
| boxedInterfaceAccountToken8 | 10205 | N/A |
| interfaceAccountMint1 | 2530 | N/A |
| interfaceAccountMint2 | 4726 | N/A |
| interfaceAccountMint4 | 9431 | N/A |
| interfaceAccountMint8 | 17709 | N/A |
| interfaceAccountToken1 | 1755 | N/A |
| interfaceAccountToken2 | 3211 | N/A |
| interfaceAccountToken4 | 6006 | N/A |
| interface1 | 999 | N/A |
| interface2 | 1574 | N/A |
| interface4 | 1996 | N/A |
| interface8 | 3651 | N/A |
| program1 | 999 | N/A |
| program2 | 1573 | N/A |
| program4 | 1998 | N/A |
| program8 | 3651 | N/A |
| signer1 | 958 | N/A |
| signer2 | 1576 | N/A |
| signer4 | 2079 | N/A |
| signer8 | 3895 | N/A |
| systemAccount1 | 1013 | N/A |
| systemAccount2 | 1686 | N/A |
| systemAccount4 | 2298 | N/A |
| systemAccount8 | 4336 | N/A |
| uncheckedAccount1 | 953 | N/A |
| uncheckedAccount2 | 1567 | N/A |
| uncheckedAccount4 | 2060 | N/A |
| uncheckedAccount8 | 3855 | N/A |

---
15 changes: 15 additions & 0 deletions tests/bench/Anchor.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[provider]
cluster = "localnet"
wallet = "~/.config/solana/id.json"

[programs.localnet]
bench = "Fg6PaFpoGXkYsidMpWTK6W2BeZ7FEfcYkg476zPFsLnS"

[workspace]
members = ["programs/bench"]

[scripts]
test = "yarn run ts-mocha -t 1000000 -p ./tsconfig.json -t 1000000 tests/**/*.ts"
update-bench = "yarn run ts-node scripts/update-bench.ts"
generate-ix = "yarn run ts-node scripts/generate-ix.ts"
bump-version = "yarn run ts-node scripts/bump-version.ts"
4 changes: 4 additions & 0 deletions tests/bench/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
[workspace]
members = [
"programs/*"
]
27 changes: 27 additions & 0 deletions tests/bench/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Benchmark tests

The bench program and its tests are used to measure the performance of Anchor programs.

## How

Create a program -> Write tests that measure usage -> Compare the results -> Save the new result

The script will check whether there is a difference between the current result and the last saved result(in `bench.json`) at the end of the tests. If the difference between the results is greater than 1%, the new data will be saved in `bench.json` and Markdown files in [/bench](https://github.com/coral-xyz/anchor/tree/master/bench) will be updated accordingly.

## Scripts

`anchor test --skip-lint`: Run all tests and update benchmark files when necessary. This is the only command that needs to be run for most use cases.

---

The following scripts are useful when making changes to how benchmarking works.

`anchor run update-bench`: Update Markdown files in [/bench](https://github.com/coral-xyz/anchor/tree/master/bench) based on the data from `bench.json`.

`anchor run generate-ix`: Generate instructions with repetitive accounts.

---

The following script is only for the maintainer(s) of Anchor.

`anchor run bump-version -- <VERSION>`: Bump the version in all benchmark files.
Loading