Latest Mobile VO fails to read out responsive descriptions

[Nancy-Salpepi]

Test device
iPad 9th generation

Operating System
iPadOS 17.5.1

Browser
Safari

Problem description
Seen while testing phetsims/qa#1086 but also happens in published:

Mobile VO no longer reads out all the responses when an action is taken (a button is pressed or a slider is moved). @KatieWoe confirmed that she is also seeing the same thing with her iPad and iPadOS 17.5. It seems like Apple broke something in 17.5.

For example, in Greenhouse Effect, go to the Waves Screen and press Start Sunlight -- you will not hear "warming" or any temperature values. When using the concentration slider it only indicates the level of greenhouse gases. The same thing happens in the published version.

I also get no responses with Friction in published (and main)--it doesn't say that the book is grabbed or released or give any description about what is happening to the atoms as the books are rubbed together.

Published GFLB and RAP give some responses.

[jessegreenberg]

Bummer - OK thanks for reporting @Nancy-Salpepi.

Published GFLB and RAP give some responses.

Can you please clarify, does that mean on iPadOS 17.5 some responses are spoken successfully? Can you give an example of the ones you do hear and the ones you don't hear for those sims?

[KatieWoe]

For RaP, when moving the hands it will say, "close to target" or similar, but it won't read out "hand on tick 4" or things like that when it should.

[jessegreenberg]

OK, that is helpful thanks. It sounds like we are hearing the aria-valuetext but none of the aria-live alerts.

[jessegreenberg]

I just tried on iOS 15.0.2 and I heard alerts as expected in greenhouse-effect. Ill see if I an try a newer version.

[jessegreenberg]

I was also able to test on 16.5 and confirmed alerts sound correct.

[jessegreenberg]

@KatieWoe did the following test:

• Go to http://htmlpreview.github.io/?https://github.com/phetsims/a11y-research/blob/main/reader-intro.html
• Turn on VO
• Press the "Create random fact!" button
  VoiceOver read the aria-live alert as expected. So something else isn't working.

I am able to install iOS 17.5.1 on an iPad, I am doing that now to test.

[jessegreenberg]

After updating to 17.5.1 I am observing this problem.

[jessegreenberg]

If I remove this, the problem goes away. However, VoiceOver interrupts itself for every queued up aria-live=polite alert and we only hear the most recent information.

utterance-queue/js/AriaLiveAnnouncer.ts

Lines 234 to 242 in a2b10ee

	if ( platform.safari ) {
	// Using `hidden` rather than clearing textContent works better on mobile VO,
	// see https://github.com/phetsims/scenery-phet/issues/490
	liveElement.hidden = true;
	}
	else {
	liveElement.textContent = '';
	}

If we can fix this, it will be in utterance-queue so transfering the issue there.

[jessegreenberg]

aria-live='polite' does not queue at all in iOS VoiceOver 17.5.1, as seein in this example:

<h1>Testing</h1>
<p id='alerter' aria-live='polite'></p>
<p id='alerter2' aria-live='polite'></p>

<script>

  // alternate between two alert elements (similar to how UtteranceQueue works)
  const alerter = document.getElementById( 'alerter' );
  const alerter2 = document.getElementById( 'alerter2' );

  let counter = 1;
  window.setInterval( () => {
    if ( counter % 2 === 0 ) {
      alerter2.textContent = `This is a new alert that is quite long, will it be interrupted? ${counter}`;
    }
    else {
      alerter.textContent = `This is a new alert that is quite long, will it be interrupted? ${counter}`;
    }

    counter++;
  }, 1000 );
</script>

[jessegreenberg]

The above commit improves the output on iOS VoiceOver. But it has a negative impact on MacOS VoiceOver. It breaks the queuing of aria-live=polite alerts for both platforms.

Discussing with @Nancy-Salpepi and we think the best thing to do is to apply that change for iOS VoiceOver but NOT change MacOS VoiceOver.

Note this will improve iOS VoiceOver but we will still miss a lot of alerts as they get interrupted.

Done in 58282ac.

We should discuss is this change is good enough and if it requires a maintenance release.

EDIT: I noted in the planning slack channel that this issue, along with phetsims/scenery#1631, will require some maintenance time. I will work there to coordinate discussion and maintenance.

[kathy-phet]

Bummer! :( But glad you found a bit of an improvement.

[terracoda]

I think we need to reach out to apple to check on its support for ARIA-Live.

[jessegreenberg]

I agree. The specification here (https://www.w3.org/TR/wai-aria-1.2/#aria-live) describes that polite alerts should queue. (Though to be fair, it leaves room for interpretation). #117 (comment) demonstrates that they do not. That is sufficient for a bug report, which we can submit, but we have never received a response from Apple's Feedback Assistant.

[terracoda]

Ok, I sent an email with the question jesse suggested:

Our Question:
Why doesn't VoiceOver queue alerts from aria-live=polite? The ARIA spec suggests they should queue (see https://www.w3.org/TR/wai-aria-1.2/#aria-live). Other screen reader vendors implement a queue for aria-live=polite alerts. VoiceOver users are missing important information in web applications because alerts do not queue.

[terracoda]

I heard back from Chris Fleizach.

They would very much appreciate a webkit bug report.

Filing a webkit bug will be very much appreciated
Thank you

[KatieWoe]

Making a note that we still see this problem with Mobile VO on iPadOS 18

[terracoda]

@jessegreenberg mentioned we can follow-up on this issue and submit an issue to webkit as we work on description for pH Scale: Basics.