In iPadOS PhET sims can be interacted with with bluetooth keyboard - but only partially

[Ashton-Morris]

Jessie and I were seeing today if you could use alt keyboard input on iPad to control the sim using a bluetooth keyboard.

We found that there is a setting which allows this to be possible. After reloading the page (safari) I was able to navigate and use the sim with about 70% of the interactions working as usual as on a computer. Its a little clunky compared to Mac OS so there may be something causing issues with the keyboard navigation.

Here is the seeing that allows navigation.

[jessegreenberg]

Thanks @Ashton-Morris , I am going to move to scenery. I suspect this is the same problem we saw many years ago where keyup events are not sent to Safari. But we don't have enough evidence to prove it yet and couldn't get enough info from tethering the device.

Ill try out on an ipad if I can find one, otherwise I will create a test harness to pair with someone with an iPad to get more info.

[jessegreenberg]

I built version of a sim with some debugging information that should appear on screen since tethering wasn't working very well. @Nancy-Salpepi and I will meet today to test on an iPad with a bluetooth keyboard.

https://phet-dev.colorado.edu/html/jg-tests/quadrilateral_en_phet.html

[jessegreenberg]

Testing with @Nancy-Salpepi:

With default settings, we saw the pink focus higlight around vertices/sides and could move them well with the keyboard. However, we could not navigate to any other "basic" controls like buttons/checkboxes.

Then, we changed the setting @Ashton-Morris described in #1659 (comment). This allowed us to tab to the basic basic controls and use them with a keyboard.

However, this made it impossible to move the vertices/sides with the keyboard. Instead, the arrow keys navigated around in the address bar. It also prevented focus from being in the sim on startup. We also did not get keyup events on tab press or when you activate buttons/checkboxes.

The lack of keyup explains why we do not see the focus highlight when you tab into the sim.

We also tested on http://htmlpreview.github.io/?https://github.com/phetsims/a11y-research/blob/main/reader-intro.html and saw the same results (outside of a PhET sim):

With the navigation setting turned OFF, you cannot interact at all with the page with a keyboard.
With the navigation setting turned ON, you can interact with the basic web components.

[jessegreenberg]

The TL;DR for bluetooth keyboards with iPadOS:

With default settings, you can tab to custom components and interact with them with a keyboard, but you cannot navigate to other basic form components.

With "Full Keybord Access" accessibility setting turned on, you can navigate and interact with basic form components, but you cannot use custom interactive components (anything that uses arrow keys). Often, our default pink focus highlight will not appear.

[jessegreenberg]

@Nancy-Salpepi added a note in the QA documentation for this. I added a note in joist about why keyboard help is hidden on that platform for now.

I sent this to the design-sims slack channel:

We learned some information about how iPadOS works with Bluetooth keyboards. #1659 (comment). The behavior for alt input is different. I don't know if this combination gets used much, but thought people should be aware. If we expect this combo to be used, should this info go anywhere so teachers can find it?

Closing the scenery issue because we understand this behavior now.