[forge] add indexer_test to unstable; clean up unused forge workflows

[rustielin]

Description

Depends on #14851

This PR adds the new indexer_test testsuite to "Forge Unstable", where it will bake for a few days, after which point we will have enough data to tune the workload and latency success criteria thresholds, and then add it to "Forge Stable". To prevent old/stale jobs from clogging up "Forge Unstable", start fresh by removing all existing jobs from there and make it run continuously at a relatively quick cadence (every 2 hrs).

Also:

• Remove the "Forge PFNs" workflow, as it has not been run in a while
• Remove the "Forge State Sync" workflow, as it has not been run in a while
• Copy a lot of the boilerplate from "Forge Stable" to "Forge Unstable". We should really consider generating these workflows as we do with deployment automation actually, though with only 2 workflows today it is at least a bit manageable.

How Has This Been Tested?

See the dependent PR's test plan for the indexer test suite

For the GHA workflow, validation on this PR should be sufficient.

Key Areas to Review

The cleaned up workflows, and also the new cadence.

Type of Change

• New feature
• Bug fix
• Breaking change
• Performance improvement
• Refactoring
• Dependency update
• Documentation update
• Tests

Which Components or Systems Does This Change Impact?

• Validator Node
• Full Node (API, Indexer, etc.)
• Move/Aptos Virtual Machine
• Aptos Framework
• Aptos CLI/SDK
• Developer Infrastructure
• Move Compiler
• Other (specify)

Checklist

• I have read and followed the CONTRIBUTING doc
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I identified and added all stakeholders and component owners affected by this change as reviewers
• I tested both happy and unhappy path of the functionality
• I have made corresponding changes to the documentation

[trunk-io]

⏱️ 45m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
dispatch_event	15m	🟩
rust-doc-tests	5m	🟩
execution-performance / test-target-determinator	4m	🟩
test-target-determinator	4m	🟩
check	4m	🟩
rust-cargo-deny	3m	🟩 🟩
rust-move-tests	2m	🟩
rust-move-tests	2m	🟩
fetch-last-released-docker-image-tag	1m	🟩
check-dynamic-deps	1m	🟩 🟩
general-lints	1m	🟩 🟩
semgrep/ci	45s	🟩 🟩
file_change_determinator	24s	🟩 🟩
execution-performance / single-node-performance	11s	🟩
file_change_determinator	10s	🟩

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
execution-performance / single-node-performance	11s	18m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions]

✅ Forge suite compat success on 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 798df1ddee51e2e09c2c86898d9443646c4dc061

Compatibility test results for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 798df1ddee51e2e09c2c86898d9443646c4dc061 (PR)
1. Check liveness of validators at old version: 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775
compatibility::simple-validator-upgrade::liveness-check : committed: 15788.09 txn/s, latency: 2127.48 ms, (p50: 2100 ms, p70: 2200, p90: 2500 ms, p99: 3900 ms), latency samples: 516620
2. Upgrading first Validator to new version: 798df1ddee51e2e09c2c86898d9443646c4dc061
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7632.24 txn/s, latency: 3791.16 ms, (p50: 4300 ms, p70: 4400, p90: 4500 ms, p99: 4600 ms), latency samples: 143980
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6614.90 txn/s, latency: 4835.00 ms, (p50: 5000 ms, p70: 5100, p90: 6800 ms, p99: 7300 ms), latency samples: 221500
3. Upgrading rest of first batch to new version: 798df1ddee51e2e09c2c86898d9443646c4dc061
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6735.42 txn/s, latency: 4237.01 ms, (p50: 4800 ms, p70: 5000, p90: 5200 ms, p99: 5300 ms), latency samples: 127480
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6901.21 txn/s, latency: 4620.93 ms, (p50: 5200 ms, p70: 5300, p90: 5700 ms, p99: 6100 ms), latency samples: 233720
4. upgrading second batch to new version: 798df1ddee51e2e09c2c86898d9443646c4dc061
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10390.18 txn/s, latency: 2596.78 ms, (p50: 2500 ms, p70: 2700, p90: 4400 ms, p99: 4700 ms), latency samples: 190640
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10508.24 txn/s, latency: 2927.76 ms, (p50: 2600 ms, p70: 2800, p90: 5500 ms, p99: 7300 ms), latency samples: 340320
5. check swarm health
Compatibility test for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 798df1ddee51e2e09c2c86898d9443646c4dc061 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite realistic_env_max_load success on 798df1ddee51e2e09c2c86898d9443646c4dc061

two traffics test: inner traffic : committed: 12106.21 txn/s, latency: 3288.14 ms, (p50: 3000 ms, p70: 3300, p90: 3900 ms, p99: 6300 ms), latency samples: 4603040
two traffics test : committed: 100.01 txn/s, latency: 2983.74 ms, (p50: 2600 ms, p70: 3100, p90: 3700 ms, p99: 10700 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.254, avg: 0.231", "QsPosToProposal: max: 0.441, avg: 0.388", "ConsensusProposalToOrdered: max: 0.333, avg: 0.329", "ConsensusOrderedToCommit: max: 0.499, avg: 0.485", "ConsensusProposalToCommit: max: 0.831, avg: 0.814"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.77s no progress at version 7939 (avg 0.23s) [limit 15].
Max epoch-change gap was: 1 rounds at version 2452626 (avg 1.00) [limit 4], 9.40s no progress at version 2452626 (avg 9.40s) [limit 15].
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions]

✅ Forge suite framework_upgrade success on 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 798df1ddee51e2e09c2c86898d9443646c4dc061

Compatibility test results for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 798df1ddee51e2e09c2c86898d9443646c4dc061 (PR)
Upgrade the nodes to version: 798df1ddee51e2e09c2c86898d9443646c4dc061
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1278.20 txn/s, submitted: 1280.22 txn/s, failed submission: 2.02 txn/s, expired: 2.02 txn/s, latency: 2421.68 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5000 ms), latency samples: 113740
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1135.10 txn/s, submitted: 1138.94 txn/s, failed submission: 3.85 txn/s, expired: 3.85 txn/s, latency: 2853.40 ms, (p50: 2400 ms, p70: 2700, p90: 5100 ms, p99: 7300 ms), latency samples: 100360
5. check swarm health
Compatibility test for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 798df1ddee51e2e09c2c86898d9443646c4dc061 passed
Upgrade the remaining nodes to version: 798df1ddee51e2e09c2c86898d9443646c4dc061
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1138.92 txn/s, submitted: 1140.28 txn/s, failed submission: 1.35 txn/s, expired: 1.35 txn/s, latency: 2747.43 ms, (p50: 2600 ms, p70: 3000, p90: 4400 ms, p99: 6800 ms), latency samples: 100980
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[sionescu]

This should still be triggered by forge-unstable.yaml.

[sionescu]

Use the [[ operator.