Support for alternative input

[pixelzoom]

@kathy-phet would like to add alternative input to this sim, and asked me to provide some thoughts. My understanding is that he'd wants to have a discussion, sometime towards the beginning of Q2 2022. So assigning to @kathy-phet and @amanda-phet.

[pixelzoom]

As with UI sound #181, there's very little alternative input that will come for free with this sim. This sim doesn't use many standard UI components -- a handful of push buttons, radio buttons, number pickers, and accordion boxes. Most of the interaction occurs with things that are custom to this sim.

The first heavy lift will be design, involving many people, a big job for someone to lead.

The second heavy lift will be implementation. It will certainly require both sim-specific and common-code work. There are common-code aspect of a11y that are still experiencing significant problems (e.g hotkeys, pan/zoom) and there will be a need for new common-code support. There will be a need for sim-specific code.

Some of the bigger questions that we’ll need to answer, not a complete list:

• Mouse/touch and keyboard currently require separate drag handlers, and they differ in some ways (see GrabDragInteraction issues when combining mouse/touch and alternative input scenery-phet#710). Should PhET put some work into unifying more of the mouse/touch/keyboard dragging in scenery before tackling this sim?

• This sim has some incredibilty complicated DragListeners, and is already pushing the limits for what is understandable and maintainable. For every DragListener, we’ll need to add a KeyBoardDragListener. So factoring out drag code and adding the KeyboardDragListeners is going to be a very unenviable task, highly likely to break the sim. Even determining whether dragging is broken is going to be difficult - recall that this sim has an entire document just to describe the lock scenarios. PhET definitely runs the risk of pushing this sim over this edge in terms of complexity.

• A general design of alt input for toolboxes doesn't currently exist. We got partway there in Geometric Optics (see Alt input for rulers and toolbox. geometric-optics#258), realized how difficult the problem was, and took a simplified approach -- we skipped grab-drag interactions and just used focus traversal to select tools. That simplified approach is not going to work for Equality Explorer. We’ll need grab-drag interaction, and the general design will need to be completed, see Develop a design pattern for Alt input of toolboxes/creator pattern a11y-research#166.

• We’ll definitely need grab-drag interaction for interacting with items. If an item is not grabbed (help onto), it will animate to the toolbox or scale. Will the grab-drag interaction UI used for the balloon in BASE work here? The balloon is a much larger object.

• Each plate of the scale can hold up to 36 items. What is the UX for interacting with those items? How do we select which item we want to grab on the scale? We certainly don’t want to tab through 72 individual items to get to the one we’re interested in, or to some UI component that is before/after the scale in traversal order. Is the entire group of items one focusable thing, where we can then move around inside that thing (with arrow keys?) to choose the item we want to grab? The decisions made here may necessitate some major changes to the structure of the code, depending on how focus works, and how we select items.

• Where would hotkeys be useful?

Finally... Equality Explorer is the most complicated sim that I've implemented for PhET, but has been remarkably stable and bug-free. Having to make any changes to it gives me the willies. Having to make the kind of changes required to support alt input?... oh boy. And there are some brand new, really big design problems, with no common-code support. Are you sure this is where you want to experiment with those problems?

[pixelzoom]

According to @kathy-phet in Slack, "we are going to postpone that work on EE for now". So unassigning myself, and labeling as deferred.