UI Sounds and focus - consider earcons for Voiced experience

[terracoda]

The screen reader experience contains spoken document and interaction semantics, and some screen readers even have some non-speech sounds that indicate certain things - like an element is interactive. These non-speech sounds may be customizable in some way, I am not up to speed on how they work.

Our Voicing feature, does not directly target blind learners, but we do not not necessarily exclude blind learners from a reasonably good experience - especially in a collaborative contexts where learners with and without vision are learning together.

I am wondering if we have considered the use UI sounds that could potentially fire on focus events and thus could potentially communicate something about the object - something as general as it is interactive, or something more specific, like its a button, checkbox, slider, or custom draggable object?

We have a library of UI sounds for activation events - pressing a button, checking a checkbox, picking up a draggable object. I am wondering if we have considered using any non-speech sounds (i.e. earcons) that could fire on focus events?

The reason I am asking is that as the Voicing feature becomes more popular, it might be used by more learners who may be used to hearing spoken semantics. Focused-based earcons might be helpful addition to the Voice experience.

[emily-phet]

@terracoda I'm not aware of this coming up before.
A few follow up questions;

• Would the sounds you're imagining be on by default when Voicing is turned on?
• How many different sounds would you imagine being needed? It seems like everything that has focus is interactive (though I guess reading blocks and disabled objects are not...), and having a different sound for each sim component type seems perhaps too much for a person to distinguish (even with great sound design, of course!). So maybe there's a small set of categories we could flush out pretty quickly - e.g., a standard sound as focus moves, and then a few custom variants for specialty items (like reading blocks, etc.).

Since most sims are intentionally designed to be "shallow" with respect to depth - we want people to feel comfortable pushing buttons and seeing what happens, and not worrying about 'messing up' anything, my initial thought is that an interactive vs non-interactive sounds would be helpful, but distinguishing too much further than that would have diminishing returns for users in a situation where the focus is best on "play with it all" rather than taking energy and attention to figure out what everything is semantically. Maybe interactive, non-interactive, and then a variant of the interactive sound that plays when focus lands on a custom object, sense those objects are quite unique (three sounds total).

[jbphet]

The general idea of focused-based earcons seems quite interesting to me, and I'd be happy to build prototypes and try things out. I'm not sure where this would land in terms of my priorities though, and at the moment I am quite limited in the amount of time for a11y in general (20%).

Assigning to @terracoda to answer @emily-phet's questions above, and to @emily-phet to set the priority of any sound design and/or development effort to investigate this idea.

[emily-phet]

This would be great, but there's no time for it now. Marking as deferred.