VoiceOver bot completely incorrect for Tab key output

[jha11y]

Description of Behavior

For the Tab key output for test 6 from the VoiceOver bot was:

Leaving Radio Group Example Using Roving tabindex. Entering toolbar. show sidebar button
Show sidebar You are currently on a button, inside of a toolbar. To click this button, press Control -Option -Space. To navigate items on this toolbar, press Control -Option, and then Up Arrow, Down Arrow, Left Arrow, or Right Arrow. To exit this toolbar, press Control -Option -Shift -Up Arrow.

Test Setup

• Test File: test-11-navOutEndRadioGroup-voiceover_macos.collected.html
• AT: VoiceOver for macOS (Version 15.1)
• Browser: Safari (Version 18)

[outofambit]

Hi @jha11y, thanks for sharing this. Are you able to share a link to the test run on the ARIA-AT app by any chance? From my investigation it looks like the VoiceOver bot is likely using the wrong version of macOS when 15.x is specified. Thank you!

[outofambit]

Hi again @jha11y I'v confirmed that the app previously allowed users to run the bot with macOS 15 even though its not supported. So the output you received was actually from the macOS 13 version of VoiceOver and therefore doesn't match what you'd see locally on macOS 15.

We tracked that down to an issue with the release date that was incorrectly entered for VoiceOver 15.1. I've corrected that release date, so the app should accurately reflect which versions of VoiceOver can be run with the bot currently. We are working to support macOS 15, but are currently blocked by GitHub Actions' lack of support for VoiceOver configuration.

I've also opened a PR to add a little date validation to help prevent the data mis-entry that happened from happening in the future.

Please let me know if you have any questions!