Skip to content

Conversation

Elvis339
Copy link
Contributor

@Elvis339 Elvis339 commented Sep 18, 2025

Why this should be merged

This patch introduces a custom GitHub Actions workflow that enables automated performance testing and comparison of two arbitrary versions of Firewood running C-Chain re-execution benchmarks on our self-hosted runners. This capability is essential for:

  • Performance regression detection: Establishes baseline comparison capability with statistical analysis coming in follow-up PR for more rigorous regression identification
  • Release validation: Compare candidate releases against stable baselines before deployment

How this works

The workflow implements an A/B testing approach with the following architecture:

  1. Runs baseline and candidate versions simultaneously on identical hardware
  2. Flexible Version Support:
    • Tagged releases (uses pre-built binaries when available for faster setup)
    • Commit hashes (builds from source for development testing)
    • Compatible AvalancheGo version pairing for each Firewood version
  3. Automated Comparison: Extracts performance metrics (mgas/s) and calculates percentage changes

Technical Implementation

This workflow enables configuration options:

  • Runner Selection
    • Choose optimal hardware for consistent benchmarking
  • Version Pairing Strategy
    • The workflow requires specifying both Firewood and AvalancheGo versions for each test case, ensuring compatibility matrix (AvGO <--depends-on-> CorETH -> Firewood)
  • Baseline Pair: Known stable versions for comparison reference
    • firewood-baseline-version: Stable Firewood release (e.g., ffi/v0.0.12) or commit hash
    • avalanchego-baseline-version: Compatible AvalancheGo version (defaults to master)
  • Candidate Pair: Versions under test
    • firewood-candidate-version: New Firewood version to evaluate
    • avalanchego-candidate-version: Corresponding AvalancheGo version
  • Benchmark Scope Control
    • Block Range: Configurable test boundaries (start-block to end-block)
    • Default Range: Blocks 101-200 provides a quick ~100-block validation

How this was tested

Ran in CI

Need to be documented in RELEASES.md?

No. This is an internal development tool that doesn't affect end-user functionality or require release notes.

Compares performance between two Firewood versions by running C-Chain re-execution benchmarks.
Supports both tagged releases (uses pre-built binaries when available) and source builds.
Each version runs with its specified AvalancheGo commit for compatibility.
@Elvis339 Elvis339 changed the title C chain reexecution firewood ab test ci: ab test firewood in reexecution benchmarks Sep 18, 2025
@Elvis339 Elvis339 self-assigned this Sep 19, 2025
@joshua-kim joshua-kim moved this to Ready 🚦 in avalanchego Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready 🚦
Development

Successfully merging this pull request may close these issues.

2 participants