test json diff workflow #14806

yuunlimm · 2024-09-30T21:35:57Z

Description

This is a new workflow that will trigger a processor integration test when it's needed.

The workflow is triggered manually (workflow_dispatch) or when a pull request is created/updated on the main branch.
The workflow runs a CLI to generate transaction proto json files based on the config/ move fixtures.
The json files are cleaned by removing specific fields that are non-deterministic from the scripted transcations for better comparison (e.g., timestamps, addresses).
We compare newly generated files from the workflow with the existing files under ecosystem/indexer-grpc/indexer-test-transactions/json_transactions
If new files are detected, it stops the process, requiring manual review.
If differences are found, it proceeds with dispatching an event.
If no differences are found, we successfully stops the process.
When differences are detected, it dispatches an event to the aptos-indexer-processors repository to trigger related processor tests.
- After dispatching the event, the workflow polls the status of the triggered tests in the aptos-indexer-processors repo.
- If the tests complete successfully, the workflow finishes with success. Otherwise, it retries up to a certain number of attempts or exits with failure if the job does not complete.

How Has This Been Tested?

when new field is added:

when no new files: https://github.com/aptos-labs/aptos-core/actions/runs/11225312911/job/31203769923

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

I have read and followed the CONTRIBUTING doc
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I identified and added all stakeholders and component owners affected by this change as reviewers
I tested both happy and unhappy path of the functionality
I have made corresponding changes to the documentation

trunk-io · 2024-09-30T21:36:00Z

⏱️ 16h 55m total CI duration on this PR

Slowest 15 Jobs	Cumulative Duration	Recent Runs
dispatch_event	4h 31m	🟥 🟥 🟥 🟥 🟥 (+22 more)
rust-cargo-deny	2h 13m	🟩 🟩 🟩 🟩 🟩 (+75 more)
check-dynamic-deps	1h 30m	🟩 🟩 🟩 🟩 🟩 (+76 more)
general-lints	37m	🟩 🟩 🟩 🟩 🟩 (+75 more)
semgrep/ci	32m	🟩 🟩 🟩 🟩 🟩 (+75 more)
dispatch_event	20m	🟥
dispatch_event	16m	🟩
file_change_determinator	14m	🟩 🟩 🟩 🟩 🟩 (+75 more)
dispatch_event	12m	🟩 🟩
dispatch_event	11m	🟥
dispatch_event	11m	🟥
dispatch_event	10m	🟥
dispatch_event	10m	🟥
dispatch_event	10m	🟥 🟥
dispatch_event	10m	🟩

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

larry-aptos · 2024-10-08T17:24:29Z

.github/workflows/indexer-processor-testing.yaml

+      # TODO: improve this step to be easily maintainable and extensible
+      # Prepare Original and New JSON Files
+      - name: Prepare and Clean JSON Files
+        run: |


i feel we can use python: managing or ignoring some fields are easier.

larry-aptos

approve to unblock; leaving some comments to follow up.

.github/workflows/indexer-processor-testing.yaml

yuunlimm · 2024-10-08T19:02:20Z

github-actions · 2024-10-08T20:43:22Z

✅ Forge suite `compat` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `0ffb969b024585a74be0c49ed2e0256d1774ffe4`

Compatibility test results for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 0ffb969b024585a74be0c49ed2e0256d1774ffe4 (PR)
1. Check liveness of validators at old version: 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775
compatibility::simple-validator-upgrade::liveness-check : committed: 11723.46 txn/s, latency: 2835.38 ms, (p50: 1800 ms, p70: 2000, p90: 3100 ms, p99: 27800 ms), latency samples: 465880
2. Upgrading first Validator to new version: 0ffb969b024585a74be0c49ed2e0256d1774ffe4
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7532.86 txn/s, latency: 3766.46 ms, (p50: 4200 ms, p70: 4500, p90: 4600 ms, p99: 4700 ms), latency samples: 140240
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7338.45 txn/s, latency: 4412.23 ms, (p50: 4700 ms, p70: 4800, p90: 5900 ms, p99: 6400 ms), latency samples: 244400
3. Upgrading rest of first batch to new version: 0ffb969b024585a74be0c49ed2e0256d1774ffe4
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 5021.09 txn/s, latency: 5828.74 ms, (p50: 6100 ms, p70: 7300, p90: 7700 ms, p99: 7900 ms), latency samples: 96600
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5397.85 txn/s, latency: 5990.45 ms, (p50: 6500 ms, p70: 6700, p90: 7300 ms, p99: 7700 ms), latency samples: 197280
4. upgrading second batch to new version: 0ffb969b024585a74be0c49ed2e0256d1774ffe4
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 10720.07 txn/s, latency: 2440.31 ms, (p50: 2600 ms, p70: 2800, p90: 2900 ms, p99: 3300 ms), latency samples: 198100
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11763.62 txn/s, latency: 2659.47 ms, (p50: 2600 ms, p70: 2800, p90: 3000 ms, p99: 3300 ms), latency samples: 379660
5. check swarm health
Compatibility test for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 0ffb969b024585a74be0c49ed2e0256d1774ffe4 passed
Test Ok

github-actions · 2024-10-08T20:44:53Z

✅ Forge suite `realistic_env_max_load` success on `0ffb969b024585a74be0c49ed2e0256d1774ffe4`

two traffics test: inner traffic : committed: 12039.35 txn/s, latency: 3302.41 ms, (p50: 3000 ms, p70: 3300, p90: 4200 ms, p99: 7800 ms), latency samples: 4577640
two traffics test : committed: 99.90 txn/s, latency: 2881.36 ms, (p50: 2500 ms, p70: 2700, p90: 4300 ms, p99: 9700 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.235, avg: 0.218", "QsPosToProposal: max: 0.551, avg: 0.447", "ConsensusProposalToOrdered: max: 0.348, avg: 0.316", "ConsensusOrderedToCommit: max: 0.497, avg: 0.473", "ConsensusProposalToCommit: max: 0.844, avg: 0.789"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.88s no progress at version 2661352 (avg 0.22s) [limit 15].
Max epoch-change gap was: 1 rounds at version 2003545 (avg 1.00) [limit 4], 9.34s no progress at version 2003545 (avg 9.34s) [limit 15].
Test Ok

github-actions · 2024-10-08T20:45:53Z

✅ Forge suite `framework_upgrade` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `0ffb969b024585a74be0c49ed2e0256d1774ffe4`

Compatibility test results for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 0ffb969b024585a74be0c49ed2e0256d1774ffe4 (PR)
Upgrade the nodes to version: 0ffb969b024585a74be0c49ed2e0256d1774ffe4
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1160.73 txn/s, submitted: 1163.01 txn/s, failed submission: 2.28 txn/s, expired: 2.28 txn/s, latency: 2630.52 ms, (p50: 2400 ms, p70: 2700, p90: 3600 ms, p99: 5800 ms), latency samples: 101740
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1224.10 txn/s, submitted: 1226.03 txn/s, failed submission: 1.93 txn/s, expired: 1.93 txn/s, latency: 2639.13 ms, (p50: 2400 ms, p70: 2900, p90: 4200 ms, p99: 6100 ms), latency samples: 101380
5. check swarm health
Compatibility test for 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 0ffb969b024585a74be0c49ed2e0256d1774ffe4 passed
Upgrade the remaining nodes to version: 0ffb969b024585a74be0c49ed2e0256d1774ffe4
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1325.48 txn/s, submitted: 1328.01 txn/s, failed submission: 2.53 txn/s, expired: 2.53 txn/s, latency: 2427.49 ms, (p50: 2100 ms, p70: 2400, p90: 3900 ms, p99: 5700 ms), latency samples: 115440
Test Ok

yuunlimm force-pushed the yuunlimm/json-diff-indexer branch 29 times, most recently from dd98903 to f645c1c Compare October 2, 2024 03:23

yuunlimm force-pushed the yuunlimm/json-diff-indexer branch from a3094e3 to b3a85bd Compare October 8, 2024 04:49

convert to bash script for readability

4980b1b

yuunlimm force-pushed the yuunlimm/json-diff-indexer branch from b3a85bd to 4980b1b Compare October 8, 2024 05:05

temp to check new file behavior

2c61b56

yuunlimm force-pushed the yuunlimm/json-diff-indexer branch 2 times, most recently from ac3153e to 1b668fb Compare October 8, 2024 06:22

add check if workflow run exists

0bc93b4

yuunlimm force-pushed the yuunlimm/json-diff-indexer branch from 1b668fb to 0bc93b4 Compare October 8, 2024 06:33

yuunlimm requested a review from larry-aptos October 8, 2024 16:38

yuunlimm added the CICD:build-images when this label is present github actions will start build+push rust images from the PR. label Oct 8, 2024

larry-aptos approved these changes Oct 8, 2024

View reviewed changes

larry-aptos reviewed Oct 8, 2024

View reviewed changes

rtso reviewed Oct 8, 2024

View reviewed changes

.github/workflows/indexer-processor-testing.yaml Show resolved Hide resolved

rtso reviewed Oct 8, 2024

View reviewed changes

.github/workflows/indexer-processor-testing.yaml Show resolved Hide resolved

rtso reviewed Oct 8, 2024

View reviewed changes

.github/workflows/indexer-processor-testing.yaml Show resolved Hide resolved

yuunlimm added 2 commits October 8, 2024 11:12

remove testing txn

fec7122

skip auth if dispatch isn't needed

0ffb969

yuunlimm force-pushed the yuunlimm/json-diff-indexer branch from 1449fd0 to 0ffb969 Compare October 8, 2024 18:37

rtso approved these changes Oct 8, 2024

View reviewed changes

yuunlimm enabled auto-merge (squash) October 8, 2024 20:17

This comment has been minimized.

Sign in to view

yuunlimm merged commit 8bb94b7 into main Oct 8, 2024
94 checks passed

yuunlimm deleted the yuunlimm/json-diff-indexer branch October 8, 2024 20:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test json diff workflow #14806

test json diff workflow #14806

yuunlimm commented Sep 30, 2024 •

edited

Loading

trunk-io bot commented Sep 30, 2024 •

edited

Loading

larry-aptos Oct 8, 2024 •

edited

Loading

larry-aptos left a comment

yuunlimm commented Oct 8, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

github-actions bot commented Oct 8, 2024

test json diff workflow #14806

test json diff workflow #14806

Conversation

yuunlimm commented Sep 30, 2024 • edited Loading

Description

How Has This Been Tested?

Key Areas to Review

Type of Change

Which Components or Systems Does This Change Impact?

Checklist

trunk-io bot commented Sep 30, 2024 • edited Loading

larry-aptos Oct 8, 2024 • edited Loading

Choose a reason for hiding this comment

larry-aptos left a comment

Choose a reason for hiding this comment

yuunlimm commented Oct 8, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Oct 8, 2024

✅ Forge suite compat success on 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 0ffb969b024585a74be0c49ed2e0256d1774ffe4

github-actions bot commented Oct 8, 2024

✅ Forge suite realistic_env_max_load success on 0ffb969b024585a74be0c49ed2e0256d1774ffe4

github-actions bot commented Oct 8, 2024

✅ Forge suite framework_upgrade success on 46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775 ==> 0ffb969b024585a74be0c49ed2e0256d1774ffe4

yuunlimm commented Sep 30, 2024 •

edited

Loading

trunk-io bot commented Sep 30, 2024 •

edited

Loading

larry-aptos Oct 8, 2024 •

edited

Loading

✅ Forge suite `compat` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `0ffb969b024585a74be0c49ed2e0256d1774ffe4`

✅ Forge suite `realistic_env_max_load` success on `0ffb969b024585a74be0c49ed2e0256d1774ffe4`

✅ Forge suite `framework_upgrade` success on `46bf19eb4f132b9d8fc19eff3f3334cdf9aa1775` ==> `0ffb969b024585a74be0c49ed2e0256d1774ffe4`