ci build errors due to cancelled jobs

[reubenmiller]

Describe the bug

Recent failures on the merge queue show unexpected behaviour when a build job is cancelled. This resulted in the workflow continuing even though one architecture failed resulting in only half the packages being published.

Runs with unexpected behaviour

• https://github.com/thin-edge/thin-edge.io/actions/runs/9298357614/workflow - Publish was called even though one build job was cancelled
• https://github.com/thin-edge/thin-edge.io/actions/runs/9290324680/attempts/1 - job was cancelled

To Reproduce

Expected behavior

Screenshots

Environment (please complete the following information):

• OS [incl. version]
• Hardware [incl. revision]
• System-Architecture [e.g. result of "uname -a"]
• thin-edge.io version [e.g. 0.1.0]

Additional context

[reubenmiller]

A PR was merged to add an additional dependency to the build job, and the subsequent behaviour will be monitored.

[reubenmiller]

There is an active discussion going on for similar symptoms to:

The hosted runner encountered an error while running your job. (Error Type: Failure).

The above was taken from this run:

[reubenmiller]

Root cause

There is an active Github Issue where other projects have also reported similar issues with Github Workflows.

There seems to be a problem with the Github Runner which results in jobs sporadically being set to skipped, due to unknown reasons. The error can manifest in slightly different ways, but the following are the symptoms seen in the thin-edge.io project:

• Job with a matrix build is marked as "Skipped" with the description "This job was skipped"
• An error annotation is added to the workflow run

  The hosted runner encountered an error while running your job. (Error Type: Failure).
  

Secondary effects

The root cause resulted in an unexpected side-effect where the publish job was still running after the check-build was failing was due to the usage of always() in the job's if block. The usage of always() is required as the upstream test job is conditional, and without always(), the job was also be skipped.

To handle the skip case better, the explicit check for the success result on the check-build job was added.

publish:
  name: Publish ${{ matrix.job.target }}
  if: |
    always() &&
    github.event_name != 'pull_request_target' &&
    (needs.check-build.result == 'success') &&
    (needs.test.result == 'success' || needs.test.result == 'skipped')

[reubenmiller]

The linked github issue has been closed and no more job cancellations have been observed.