Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add explicit macOS Intel+ARM64 runners #458

Merged
merged 8 commits into from
May 17, 2024
Merged

Add explicit macOS Intel+ARM64 runners #458

merged 8 commits into from
May 17, 2024

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented May 16, 2024

For quite some time, GitHub actions only supported macOS on Intel hardware.
That has changed recently: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

Interestingly, we have used macos-latest which then changed hardware underneath our feet.
I noticed, because the once-per-week 5.0.0 and 5.1.1 workflows started finding an STM Sys parallel counterexample relatively consistently.
(details below)

This PR thus adds explicit macOS Intel (macos-13) and ARM64 (macos-14) workflows, thus collecting them both in the same GitHub actions CI "backend". I'll write a separate PR to remove the macOS ARM64 runners from multicoretests-ci.

Examples of the two latest runs:

https://github.com/ocaml-multicore/multicoretests/actions/runs/9048073196/job/24860738931

version: 5.0.0
architecture: arm64
system: macosx

random seed: 139266489
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential
[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential (generating)
[✓] 1000    0    0 1000 / 1000     4.6s STM Sys test sequential

[ ]    0    0    0    0 /  200     0.0s STM Sys test parallel
[✗]  143    0    1  142 /  200    42.2s STM Sys test parallel

--- Failure --------------------------------------------------------------------

Test STM Sys test parallel failed (23 shrink steps):

                             |           
                     Mkdir ([], "ddd")   
                             |           
                  .---------------------.
                  |                     |           
          Rmdir ([], "ddd")     Rmdir ([], "ddd")   

https://github.com/ocaml-multicore/multicoretests/actions/runs/9048277550/job/24861161644

version: 5.1.1
architecture: arm64
system: macosx

random seed: 222784250
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential
[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential (generating)
[✓] 1000    0    0 1000 / 1000     3.6s STM Sys test sequential

[ ]    0    0    0    0 /  200     0.0s STM Sys test parallel
[✗]   44    0    1   43 /  200    31.8s STM Sys test parallel

--- Failure --------------------------------------------------------------------

Test STM Sys test parallel failed (26 shrink steps):

                             |           
                     Mkdir ([], "ddd")   
                             |           
                  .---------------------.
                  |                     |           
          Rmdir ([], "ddd")     Rmdir ([], "ddd")

@jmid
Copy link
Collaborator Author

jmid commented May 16, 2024

CI was failing due to

  • macOS ARM64 workflows finding a parallel Sys STM counterexample
  • the opam workflow using an outdated 5.2.0+trunk

but is otherwise green.

I've added fixes for these in f75289c and 4fda9c2

@jmid
Copy link
Collaborator Author

jmid commented May 16, 2024

This should fix #359 - a source of occasional false alarms.

@jmid
Copy link
Collaborator Author

jmid commented May 17, 2024

CI summary

In comparison, this PR's macOS-ARM64 5.2.0~beta2 and trunk workflows triggered one after 17 and 122 attempts, respectively.

Out of 38 workflows 3 failed with 1 genuine issue and 2 borderline false alarms / CI issues.

@jmid
Copy link
Collaborator Author

jmid commented May 17, 2024

In an attempt to trigger an error on the in-house silicon too, adff14c bumps the count to 2500 as already done in #304.
As this is a negative test, this should only matter to macOS ARM64 on multicoretests-ci that have been hitting the 1000 count. I can however see that this does not make a difference 🤷

Reminding myself of the macOS OCurrent setup from https://tarides.com/blog/2023-08-02-obuilder-on-macos/ I suspect the test is witnessing the difference between ZFS vs whatever the GitHub actions runners are using underneath.
Ideally the test suite should hold up against changes of the underlying file system. However

  • the GitHub runners act predictably across both macOS Intel+ARM64
  • the OCurrent one is a source of occasional false alarms in the form of counterexamples STM parallel Sys tests on macOS #359

As such, collecting the macOS runners under the same CI system seems reasonable.

@jmid
Copy link
Collaborator Author

jmid commented May 17, 2024

CI summary:

In comparison, this PR's macOS-ARM64 5.2.0~beta2 and trunk workflows triggered a counterexample after 22 and 54 attempts, respectively.

The latter two failures will disappear with the merge and deployment of ocurrent/multicoretests-ci#36

Out of 38 workflows 3 failed with 1 genuine issue and 2 borderline false alarms / CI issues.

@jmid
Copy link
Collaborator Author

jmid commented May 17, 2024

Merging this one as ocurrent/multicoretests-ci#36 has now been merged

@jmid jmid merged commit dda5ff6 into main May 17, 2024
29 of 30 checks passed
@jmid jmid deleted the macos-archs branch May 17, 2024 13:04
@jmid
Copy link
Collaborator Author

jmid commented May 17, 2024

CI summary for merge to main:

None of the 37 workflows failed 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant