Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MacOS Sequoia support for VoiceOver AppleScript automation #11257

Open
5 of 16 tasks
jlp-craigmorten opened this issue Dec 31, 2024 · 9 comments
Open
5 of 16 tasks

MacOS Sequoia support for VoiceOver AppleScript automation #11257

jlp-craigmorten opened this issue Dec 31, 2024 · 9 comments

Comments

@jlp-craigmorten
Copy link

jlp-craigmorten commented Dec 31, 2024

Description

In MacOS Sequoia (15) Apple have changed where configuration is stored for enabling AppleScript control of VoiceOver.

Specifically ~/Library/Preferences/com.apple.VoiceOver4/default.plist has been replaced by the sandboxed ~/Library/Group Containers/group.com.apple.VoiceOver/Library/Preferences/com.apple.VoiceOver4/default.plist.

This means the current code at https://github.com/actions/runner-images/blob/main/images/macos/scripts/build/configure-machine.sh#L22-L28 is no longer sufficient to allow the automation of VoiceOver on macos-15 agents.

This should be updated to something similar to:

# Update VoiceOver Utility to allow VoiceOver to be controlled with AppleScript
# by creating a special Accessibility DB file (SIP must be disabled) and
# updating the user defaults system to reflect this change.
if csrutil status | grep -Eq  "System Integrity Protection status: (disabled|unknown)"; then
    sudo bash -c 'echo -n "a" > /private/var/db/Accessibility/.VoiceOverAppleScriptEnabled'

    # New requirement for MacOS Sequoia replacing the `defaults write` below. Due to the plist being located in sandboxed location SIP or similar security controls need to be relaxed before unentitled apps/scripts can modify the value.
    sudo plutil -replace SCREnableAppleScript -bool true ~/Library/Group\ Containers/group.com.apple.VoiceOver/Library/Preferences/com.apple.VoiceOver4/default.plist
fi
defaults write com.apple.VoiceOver4/default SCREnableAppleScript -bool YES

where the pertinent addition is:

sudo plutil -replace SCREnableAppleScript -bool true ~/Library/Group\ Containers/group.com.apple.VoiceOver/Library/Preferences/com.apple.VoiceOver4/default.plist

Note: care needed to make sure that this new command is only executed for MacOS 15 (Darwin 24) agents (as presumably the file does not exist for earlier versions of the OS!)

Platforms affected

  • Azure DevOps
  • GitHub Actions - Standard Runners
  • GitHub Actions - Larger Runners

Runner images affected

  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 24.04
  • macOS 12
  • macOS 13
  • macOS 13 Arm64
  • macOS 14
  • macOS 14 Arm64
  • macOS 15
  • macOS 15 Arm64
  • Windows Server 2019
  • Windows Server 2022
  • Windows Server 2025

Image version and build link

  Image: macos-15-arm64
  Version: 20241217.490
  Included Software: https://github.com/actions/runner-images/blob/macos-15-arm64/20241217.490/images/macos/macos-15-arm64-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macos-15-arm64%2F20241217.490

REF: https://github.com/guidepup/setup/actions/runs/12561961803/job/35021706292?pr=39

Is it regression?

No

Expected behavior

VoiceOver can be automated through AppleScript.

Actual behavior

VoiceOver cannot be automated through AppleScript.

Repro steps

  1. Run macos-15 agent

  2. Start VoiceOver, e.g. execute /System/Library/CoreServices/VoiceOver.app/Contents/MacOS/VoiceOverStarter

  3. Attempt to control VoiceOver using AppleScript, e.g. execute

    -- example.applescript
    tell application "VoiceOver"
      activate
      tell vo cursor to perform action
    end tell
    /usr/bin/osascript example.applescript
  4. Observe that the script errors with example.applescript:60:74: execution error: VoiceOver got an error: vo cursor doesn’t understand the “perform action” message. (-1708) when it would normally succeed (because the dictionary isn't loaded when AppleScript control is disabled).

@pravinade
Copy link
Contributor

Hi @jlp-craigmorten,
We will look into the issue and keep you posted with updates.

Thank You.

@outofambit
Copy link

outofambit commented Jan 27, 2025

Hi @pravinade @sureshe456, is there an ETA on this fix landing? Can we expect to it be in place for GA of the macOS 15 images (#11486)? This issue makes it impossible to automate testing of VoiceOver and related accessibility features on macOS 15 with GitHub actions. Thank you!

@pravinade
Copy link
Contributor

@outofambit , we're checking into this and will keep you posted.

@pravinade
Copy link
Contributor

Hi @jlp-craigmorten , Thank you for your patience.

  1. I’ve been trying to reproduce the issue using the steps you have provided, but I wasn't able to encounter the same error you have faced. Could you please share a bit more information, such as the exact workflow file you're using or any additional context where this error might be triggered? It would help me narrow down the cause and assist you further.

  2. As you mentioned, specifically ~/Library/Preferences/com.apple.VoiceOver4/default.plist has been replaced by the sandboxed ~/Library/Group Containers/group.com.apple.VoiceOver/Library/Preferences/com.apple.VoiceOver4/default.plist. However, I can confirm that I don’t see the replacement occurring with the sandboxed version in my tests.

@jlp-craigmorten
Copy link
Author

I'll try find some time to create a public repo that reproduces the issue - I'll ping here when done! The workflow I'm using atm is quite involved and doesn't demonstrate the exact issue very well.

That is unless @outofambit has something that would make it easy to replicate and test?

@outofambit
Copy link

I'm actually seeing slightly different behavior, where the steps in our GitHub action that attempt to use defaults just hang and timeout without any logging output. @pravinade there's an example here: https://github.com/bocoup/aria-at-gh-actions-helper/actions/runs/13078336317/job/36495898619.

@jlp-craigmorten
Copy link
Author

jlp-craigmorten commented Feb 1, 2025

Hi all, I've tried to create a minimal reproduction at https://github.com/guidepup/macos-15-automation-error-demo.

Locally (without enabling AppleScript support via VoiceOver Utitlity or the suggested fix above) I get the error described in the issue when running the workflow on Sequoia:

[1] 35860
******** Launching VoiceOver from starter application ********
2025-02-01 21:13:14.061 VoiceOverStarter[35860:15759807] VoiceOverDebug: _ScreenReaderStartWithTarget with option 0
2025-02-01 21:13:14.068 VoiceOverStarter[35860:15759807] VoiceOverDebug: after boostrap_look_up succeeded
2025-02-01 21:13:14.068 VoiceOverStarter[35860:15759807] VoiceOverDebug: calling _SCRStartup returned 0
craig.morten     35860   0.1  0.0 410060000     32 s001  S     9:13PM   0:00.00 (VoiceOverStarter)
[1]+  Done                    /System/Library/CoreServices/VoiceOver.app/Contents/MacOS/VoiceOverStarter
true
VO ready.
voiceOverPerform.applescript:304:318: execution error: VoiceOver got an error: vo cursor doesn’t understand the “perform action” message. (-1708)
Failed to perform AppleScript, retrying...
voiceOverPerform.applescript:304:318: execution error: VoiceOver got an error: vo cursor doesn’t understand the “perform action” message. (-1708)
Failed to perform AppleScript, retrying...
voiceOverPerform.applescript:304:318: execution error: VoiceOver got an error: vo cursor doesn’t understand the “perform action” message. (-1708)
Failed to perform AppleScript, retrying...
voiceOverPerform.applescript:304:318: execution error: VoiceOver got an error: vo cursor doesn’t understand the “perform action” message. (-1708)
Failed to perform AppleScript, retrying...
voiceOverPerform.applescript:304:318: execution error: VoiceOver got an error: vo cursor doesn’t understand the “perform action” message. (-1708)
Failed to perform AppleScript, retrying...
^C

If I enable AppleScript support then it succeeds as expected.

Interestingly the same attempt to use AppleScript on the macos-15 runner seems to just hang rather than error - apologies a poor assumption on my steps to repro in the original description, see https://github.com/guidepup/macos-15-automation-error-demo/actions/runs/13091391299 for an example workflow which has the passing job for macos-13 and macos-14, but macos-15 just hangs...?

@pravinade
Copy link
Contributor

Hi All,

I’ve used the macos-15-xlarge image in the workflow, and VoiceOver works fine with this image. If possible, please try using the macos-15-xlarge label instead of macos-15. We are currently investigating why macos-15is hanging.

You can add this line to your code and test: os: [macos-13, macos-14, macos-15, macos-15-xlarge].

Feel free to reach out if you have any questions or need further assistance.

@jlp-craigmorten
Copy link
Author

Thanks for taking a look - will have to hold off until you resolve the macos-15 issues unfortunately, large and xlarge images aren't available on your free tier for open source.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants