BCFR-1087 Detect Finality Violation on Backfill

[dhaidashenko]

Improves detection of finality violations during backfill operation. DD
Test namespace load-ccip-d6901

[github-actions]

I see you updated files related to core. Please run pnpm changeset in the root directory to add a changeset as well as in the text include at least one of the following tags:

• #added For any new functionality added.
• #breaking_change For any functionality that requires manual action for the node to boot.
• #bugfix For bug fixes.
• #changed For any change to the existing functionality.
• #db_update For any feature that introduces updates to database schema.
• #deprecation_notice For any upcoming deprecation functionality.
• #internal For changesets that need to be excluded from the final changelog.
• #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
• #removed For any functionality/config that is removed.
• #updated For any functionality that is updated.
• #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

[github-actions]

AER Report: CI Core ran successfully ✅

aer_workflow , commit

AER Report: Operator UI CI

aer_workflow , commit , Breaking Changes GQL Check

1. Workflow triggered downstream job failed: breaking-changes-gql-check

Source of Error:

Run convictional/trigger-workflow-and-wait@f69fa9eedd3c62a599220f4d5745230e237904be	2025-01-17T12:54:21.6221837Z Checking conclusion [failure]
Run convictional/trigger-workflow-and-wait@f69fa9eedd3c62a599220f4d5745230e237904be	2025-01-17T12:54:21.6222348Z Checking status [completed]
Run convictional/trigger-workflow-and-wait@f69fa9eedd3c62a599220f4d5745230e237904be	2025-01-17T12:54:21.6222977Z Conclusion is not success, it's [failure].
Run convictional/trigger-workflow-and-wait@f69fa9eedd3c62a599220f4d5745230e237904be	2025-01-17T12:54:21.6223465Z Propagating failure to upstream job

Why: The error indicates that the downstream workflow triggered by the convictional/trigger-workflow-and-wait action did not complete successfully. The conclusion of the downstream workflow was failure, which caused the upstream job to fail as well.

Suggested fix: Investigate the logs of the downstream workflow at the provided URL to identify the specific cause of the failure. Address the issue in the downstream workflow to ensure it completes successfully.

[mateusz-sekara]

food for thought, should we track these errors using a more "observable" approach? For instance, by increasing a prometheus counter whenever pollAndSave or backup fails?

[dhaidashenko]

I'm not sure that it's really helpful to track this as a metric. Failures of pollAndSave and backup are normal (RPC failed for one of the requests), so we won't be using it on overview dashboards. We might want to introduce something of a higher level. Like the latest block successfully processed by LogPoller and compare it to one observed by HT or another component. High delta of these two values signals that PollAndSaved failed too often.
In any case it seems to be out of scope for this PR.

[reductionista]

Yeah, I think the "failed to poll and save logs due to finality violation" one is worth tracking, but as I understand it we're already tracking that indirectly because Healthy() will start returning false as soon as it happens. The other ones are fairly normal, will happen due to any sort of temporary network instability which presumably is also tracked elsewhere

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube