Numbers/comparison statements not read out on ChromeOS

[stemilymill]

Chromebook version 97.0.4692.102 (32-bit)
For phetsims/qa#771

When I click the speech button to read out the number or the comparison statement (screen 3), the sound for the button click is played but the number is not read out.

Note: after I noticed this, I updated the OS/browser, and then the number was read out twice. Then when I switched screens it did not work anymore.

ChromeOS.does.not.read.numbers.out.loud.mp4

Troubleshooting information:

!!!!! DO NOT EDIT !!!!!
Name: ‪Number Play‬
URL: https://phet-dev.colorado.edu/html/number-play/1.0.0-dev.45/phet/number-play_all_phet.html?screens=1,2,3
Version: 1.0.0-dev.45 2022-01-31 11:53:19 UTC
Features missing: applicationcache, csstransforms3d, applicationcache, touch
Flags: pixelRatioScaling
User Agent: Mozilla/5.0 (X11; CrOS aarch64 14324.80.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.102 Safari/537.36
Language: en-US
Window: 1300x629
Pixel Ratio: 1.0499999523162842/1
WebGL: WebGL 1.0 (OpenGL ES 2.0 Chromium)
GLSL: WebGL GLSL ES 1.0 (OpenGL ES GLSL ES 1.0 Chromium)
Vendor: WebKit (WebKit WebGL)
Vertex: attribs: 16 varying: 15 uniform: 1024
Texture: size: 8192 imageUnits: 16 (vertex: 16, combined: 96)
Max viewport: 8192x8192
OES_texture_float: true
Dependencies JSON: {}

[chrisklus]

Thanks @stemilymill! Would you mind checking if there's any console errors?

[stemilymill]

@chrisklus Here is all that shows up in console:


number-play_all_phet…l?screens=1,2,3:894 The AudioContext was not allowed to start. It must be resumed (or created) after a user gesture on the page. https://goo.gl/7K7WLu
number-play_all_phet…l?screens=1,2,3:894 [Violation] Added non-passive event listener to a scroll-blocking 'wheel' event. Consider marking event handler as 'passive' to make the page more responsive. See https://www.chromestatus.com/feature/5745543795965952
number-play_all_phet.html:894 [Violation] 'setTimeout' handler took 454ms
number-play_all_phet.html:894 [Violation] 'setTimeout' handler took 321ms
number-play_all_phet.html:894 [Violation] 'setTimeout' handler took 266ms
number-play_all_phet.html:894 [Violation] 'setTimeout' handler took 287ms
[Violation] 'requestAnimationFrame' handler took 160ms

​

[chrisklus]

Awesome thanks - one more question, and then it might be best to setup a zoom meeting at some point since I don't have a ChromeOS device. Can you operate the sim with the dev tools open and see if those 'setTimeout' handler took errors correspond with pressing the speech button?

[stemilymill]

All of those pop up as the sim loads, before I even click into a screen from the home screen

[chrisklus]

Got it, thank you

[chrisklus]

When meeting with @jessegreenberg, he mentioned that he has a Chromebook that he could take a look with, thanks!

[jessegreenberg]

So I just saw something pretty strange. I was testing https://phet-dev.colorado.edu/html/number-play/1.0.0-dev.45/phet/number-play_all_phet.html?screens=1,2,3 on my chromebook. At first no speech synthesis worked. So then I went to test number-play master and still no speech synthesis worked. Then I went to test published john-travoltage. When I got to the sim I heard lots of numbers from number-play, even though john-travoltage was loaded. Then I started hearing john-travoltage Voicing.

After that, all speech synthesis in number play was working as expected in the dev version and in master. I tried restarting the chromebook and chrome to get the problem to happen again, but haven't been able to since.

[jessegreenberg]

I switched tabs to search for how to find my chromebook model. When I returned to the sim, all speech synthesis is broken. I also had another tab with number-play running in the background in this case, and I wonder if this is the reason it isn't working well.

[jessegreenberg]

OK, I found a consistent way to break it.

1. Load the sim and press the button or use ?countAloud to hear voicing.
2. Switch tabs to a sim that does NOT use speech synthesis and stay on that tab for ~30 seconds.
3. Come back to the sim and try to use speech synthesis, it will be broken.

It will stay broken and it is not clear what gets speech synth to start working again. I tried to use it directly in the dev tools but that didn't work. I tried going to another sim like in #138 (comment) but that didn't work. Spamming the speech button in number-play got it to come back once but not again. Restarting chrome works sometimes but not others.

[jessegreenberg]

I logged these things around the ENGINE_WAKE_INTERVAL of SpeechSynthesisAnnouncer:
console.log( this.getSynth().speaking, queue.length, this.timeSinceWakingEngine );

I am finding that when speech synthesis is broken, this.getSynth().speaking is always true!! That is on the browser side, so there is some chrome bug related to that. It thinks that it is actively speaking. That also breaks SpeechSynthesisAnnouncer.timeSinceUtteranceEnd and SpeechSynthesisAnnouncer.readyToAnnouncer. readyToAnnounce will never be true in this case, and therefore we never get to our next Utterance in the queue. Id bet that is the crux of the problem.

[jessegreenberg]

I was able to produce this in a case outside of a PhET sim: https://codepen.io/matt-west/pen/DpmMgE

In that example speech is requested synchronously with the click event on the button. Both speaking and pending on window.speechSynthesis are true. https://developer.mozilla.org/en-US/docs/Web/API/SpeechSynthesis/pending means that the browsers internal utterance queue contains unspoken Utterances.

I tried adding these to force it to stop the synth before speaking:
this.cancelSynth();
window.speechSynthesis.pause();
window.speechSynthesis.resume();

I was just able to fix it once by calling window.speechSynthesis.cancel() in the dev tools.

EDIT: cancel() in the dev tools seems to revitalize it. I am going to try calling that when the browser window becomes active.
EDIT: Didn't work...

[jessegreenberg]

For some reason wrapping the Chromebook "engine wake" workaround with calls to cancelSynth() seems to be getting it working more predictably.

        this.cancelSynth();
        this.getSynth().speak( new SpeechSynthesisUtterance( '' ) );
        this.cancelSynth();

EDIT:
In my working copy where it is performing better I also call cancel() when the browser becomes active (speechAllowedProperty).

EDIT: More testing, it isn't working very well/consistently.

[jessegreenberg]

I was able to produce the bug in this basic HTML example

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>TEST PAGE</title>
  </head>
  <body>

  <button id="button">Press me!</button>
  </body>

<script>
  document.getElementById( 'button' ).addEventListener( 'click', () => {
    window.speechSynthesis.speak( new SpeechSynthesisUtterance( 'You just pressed me!' ) );
  } );
</script>
</html>

EDIT; Changing the script to this, by calling cancel and the speaking behind a timeout of 500 ms does NOT fix it.

  document.getElementById( 'button' ).addEventListener( 'click', () => {
    console.log( 'cancelling' );
    window.speechSynthesis.cancel();

    console.log( 'speaking' );
    window.setTimeout( () => {
      window.speechSynthesis.speak( new SpeechSynthesisUtterance( 'You just pressed me!' ) );
    }, 500 );
  } );

BUT calling window.speechSynthesis.cancel() from the dev tools DOES seem to fix things, which matches what I am seeing in the sim.

[jessegreenberg]

I am about ready to throw in the towel here. Ill check in with the team to request assistance or see if we should reconsider supporting voicing on Chromebooks. I have identified that this is a Chromebook bug and reproduced the issue in a basic HTML5 example. At the very least we should submit a bug report to google.

EDIT: Actually, I had one more thought. We could keep track of the amount of time since we all speechSynthesis.speak() and when we get the start event. If it is long enough, we can assume that the synth is "stuck" and try to make a cancel call.

[chrisklus]

@jessegreenberg thank you so much for investigating! That is super weird behavior. Let me know I could be helpful at all to pair to continue brainstorming ideas.

[chrisklus]

From design meeting:

We are okay to publish to the prototype with this issue, and would add a note in the teacher tips. @kathy-phet asked QA to check other published sims with voicing on ChromeOS.

@jessegreenberg do you think the thought in your last comment is worth investigating still?

[jessegreenberg]

Yes, sorry I haven't gotten to it yet. Regarding other published sims, I fully expect this to be an issue for them as well since I was able to observe this outside of a PhET sim.

[KatieWoe]

From what I saw on the Chromebook in the office:
Number play on master: The speaker button didn't seem to work at first. Switch tabs and coming back seemed to fix it.
John Travoltage published: Voicing worked for the most part. Switching to a different tab for a minute and coming back stopped voicing until pressing one of the description buttons, then it came back. Otherwise seemed normal from a quick check. It may have skipped the first "Voicing on" dialog, but everything after seemed to play.
Gravity Force Lab Basics published: Same as John Travoltage.

[Nancy-Salpepi]

@jessegreenberg I was testing published JT and GFL:B on Chromebook this morning and voicing was working.
With chromeOS 98 I noticed delays in 3 instances:

1. When voicing is initially turned on: I don't hear anything after pressing the toggle button. It doesn't come on until after I tab through a few things.
2. After switching to another tab and then returning (see video)
3. After leaving the sim open for a while and then tabbing or clicking on things.

For Number Play: I couldn't get voicing to work until after opening JT/GFL:B in another tab and turning on voicing ad then going back to NP. After clearing history it doesn't work again.

GFLBdelay.mov

[jessegreenberg]

Thanks @KatieWoe and @Nancy-Salpepi! This is very helpful to know.

[zepumph]

@jessegreenberg briefed me on this issue today during a11y-team dev meeting, and I had a large number of thoughts about how to pursue this issue. The vast majority of them have already been tried by @jessegreenberg, but I felt like applying a bit more rigor to the potential cross-product of possible solutions may be our best bet in finding a workaround.

First off, @jessegreenberg and I thought about the prospect of giving up on supporting Chromebook (as stated in #138 (comment)). I support you @jessegreenberg!! I was also curious if we could just let this bug be, report it to google, and still consider ourselves "supporting chromebook". @jessegreenberg mentioned that this bug really looks like our fault, and so it wouldn't be that good a situation to have it break in the wild.

Next, I feel like workaround will need to consist of three separate parts, that all may be related, and may need some cross-product testing to see what works best.

1. Detection of the error case.
   i. @jessegreenberg mentioned in Numbers/comparison statements not read out on ChromeOS #138 (comment) that we can time the space between calling speak and receiving a start event, if too long, then do something to fix it.
   ii. In step each frame, test to see if it has been too long before we got a start event.
   iii. We could listen to the browser-tab-visible Property and either check the timing above, or just assume that any time on another tab is long enough to potentially break this stuff.
2. Placement of the fix:
   i. I mentioned to @jessegreenberg that it seemed like most of the testing he was doing was around the speak that is the chromebook kickstart empty speak. This is likely important to do!
   ii. In addition, I feel like we can't forget about the actual call to speak with content, because that would be the buggiest case to handle. If that happened first, we want to make sure to apply the fix in that case.
3. The fix itself. @jessegreenberg mentioned lots of stack overflow examples and tests with calling cancel, pause/resume, etc. I would recommend really hashing out all of these, and potentially trying all of them very verbosely. Perhaps cancel is needed in some cases, and pause in others.

[jessegreenberg]

OK the behavior seems a bit better after changes in phetsims/utterance-queue#66. Here is what I expect now:

When speech fails due to switching tabs or one of the other cases pointed out in #138 (comment), speech will stay broken for ~8 seconds but then start to work again more consistently.

[jessegreenberg]

@Nancy-Salpepi or @stemilymill or @KatieWoe can you please try number-play on ChromeOS again and see if this is any better?

[KatieWoe]

This does seem better. I had trouble getting voicing when I first opened the sim, but it did seem improved otherwise.

[Nancy-Salpepi]

@jessegreenberg things look much better in NP on master! It still takes a couple of clicks to initiate an utterance, but once it starts, it works well. Switching tabs and leaving it open for a while didn't have a negative affect.

[jessegreenberg]

Awesome, thanks for checking both of you! I am going to add this to the accessibility bugs doc and consider this sufficient for now.