Submit bug report to Freedom Scientific for arrow key event handling

[jessegreenberg]

From phetsims/balloons-and-static-electricity#205 we showed that JAWS does not allow for press and hold behavior with the arrow keys due to the way the AT injects 'fake' keyup events when the the user presses and holds an arrow key. Here is an example of the events when I press and hold the right arrow key without JAWS:

And with JAWS:

We should submit a bug report to Freedom Scientific about this issue.

[jessegreenberg]

Here is a JSFiddle that exmplifies this problem that we could submit in a bug report.

https://jsfiddle.net/jk757xcz/

[jessegreenberg]

Here is a link that shows an example bug report to JAWS. It is intended for the Beta bug reports, but I imagine this style could be used for any bug report.

https://www.freedomscientific.com/Forms/JAWSBetaReport/BugReportExample

This is probably the best place to submit the report:
http://www.freedomscientific.com/Forms/QualityContact

[jessegreenberg]

Here is an a rough draft of the bug report that we might send:

//-----------------------------------------------------------------------------------------------------------------

Summary:

With Chrome 54, IE 11, and Firefox 50, JAWS fires incorrect keyup events when an arrow key is pressed down on a DOM element with the ARIA application role.

Steps to reproduce:

Open the following JSFiddle in any browser: https://jsfiddle.net/jk757xcz/5/show
Press Tab to navigate to the focusable div with the application role.
Press the arrow keys while focus is on the div with role=application.
Observe the events in the content window.
For every keydown event, JAWS adds an additional keyup event while the key is still down.

Description/Comment:

Problem: If an arrow key is pressed and held down, JAWS will add errant keyup events every time a keydown event is fired. This prevents use of interactive content which requires the keyup event to listen for when the key is released.

//-----------------------------------------------------------------------------------------------------------------

Some helpful review comments from @jobara and @terracoda:

• Use /show after the JSFiddle example - it should make it easier to see what is happening and prevent focus from getting stuck in the CSS and JS tabs.
• Clarify the use case - they need to know why this is an issue in an example, and why the keyup event is critical. Maybe even provide a link to BASE.
• See if we can reproduce the issue with a dedicated JAWS user.

[jessegreenberg]

Here is an update to the bug report based on the above recommendations:

//-----------------------------------------------------------------------------------------------------------------
Summary:

With Chrome 54, IE 11, and Firefox 50, JAWS fires incorrect keyup events when an arrow key is pressed down on a DOM element with the ARIA application role.

Steps to reproduce:

• Open the following JSFiddle in any browser: https://jsfiddle.net/jk757xcz/5/show
• Press Tab to navigate to the focusable div with the application role.
• Press the arrow keys while focus is on the div with role=application.
• Observe the events in the content window.
• For every keydown event, JAWS adds an additional keyup event while the key is still down.

Description/Comment:

Problem: If an arrow key is pressed and held down, JAWS will add errant keyup events every time a keydown event is fired. To handle complex keyboard interactions in HTML5, it is necessary to use a combination of keyup and keydown events to track which keys are down, and for how long. The method is to use "keydown" and "keyup" events to update a representative keystate object that tracks which keys are currently pressed. A game loop updates the application based on the state of this object. Since JAWS fires incorrect keyup events, it is impossible to determine which keys are pressed by the user at any given time.

Example Application: http://www.colorado.edu/physics/phet/dev/html/balloons-and-static-electricity/1.1.5-dev.1/balloons-and-static-electricity_en.html

//-----------------------------------------------------------------------------------------------------------------

[jessegreenberg]

Final draft:

//-----------------------------------------------------------------------------------------------------------------
Summary:

With Chrome 54, IE 11, and Firefox 50, JAWS fires incorrect keyup events when an arrow key is held down on a DOM element with the ARIA application role.

Steps to reproduce:

Open the following JSFiddle in any browser: https://jsfiddle.net/jk757xcz/5/show
Press Tab to navigate to the focusable div with the application role.
Press the arrow keys while focus is on the div with role=application.
Observe the events in the content window.
For every keydown event, JAWS adds an additional keyup event while the key is still down.
Description/Comment:

Problem: If an arrow key is pressed and held down, JAWS will add errant keyup events every time a keydown event is fired. To handle multiple key presses for keys that are not modifiers in HTML5, it is necessary to use a combination of "keyup" and "keydown" events to track which keys are down and for how long. The method is to use "keydown" and "keyup" events to update a representative keystate object that tracks which keys are currently pressed. A game loop updates the application based on the state of this object. Since JAWS fires incorrect keyup events, it is impossible to determine which keys are pressed by the user at any given time.

Example Application: http://www.colorado.edu/physics/phet/dev/html/balloons-and-static-electricity/1.1.5-dev.3/balloons-and-static-electricity_en.html

//-----------------------------------------------------------------------------------------------------------------

The JSFiddle example was sent to a JAWS 16 user who said she would use the example and let us know if the bug can be reproduced.

[terracoda]

@jessegreenberg
Very nice and succinct.
"JSFiddle exaple" missing an "m", but maybe that is not part of your submission.

We can see how this goes before submitting another bug report regarding the switch role - i.e., lack of fall back behaviour to role checkbox.

[jessegreenberg]

Right @terracoda, not part of the submision, but corrected in the comment, thanks!

We can see how this goes before submitting another bug report regarding the switch role

Sounds great, it will be interesting to hear the response from Freedom Scientific.

[jessegreenberg]

A JAWS user verified this issue by letting us know that they saw the following output from the above JSFiddle after pressing and holding one of the arrow keys:

• Event Type: keydown, Event keyCode: 40
• Event Type: keyup, Event keyCode: 40

This is what we saw as well, it is good to have this verified. I will submit the bug report today.

[jessegreenberg]

The bug report cannot exceed 1000 characters, so I will have to trim the report by quite a bit (600 characters)

[jessegreenberg]

Report submitted on 12/7/16 - removing assignment until we hear back.

[jessegreenberg]

Freedom Scientific responded with the following:

Thank you for contacting VFO, featuring the Freedom Scientific and Optelec brands.
While in the application, take the steps below and see if this impact your issue.

1.   Activate the settings center “JAWS key+number row 6”
   

2.   In the search edit field, type in: repeat
   

3.   Arrow down to key repeat
   

4.   Press the spacebar to uncheck this item
   

5.   Tab to okay and hit enter
   

See if this resolve your issue.
Be sure to include all previous correspondence pertaining to this matter when replying to this message so that we might betterassist you.

[jessegreenberg]

I tried the suggested user setting change. This does not fix the issue, but disables any events from firing while the arrow keys are pressed and held down. WASD keys work as usual regardless of this setting. In the JSFiddle example (https://jsfiddle.net/jk757xcz/5/show) if I press and hold an arrow key I see one key down event and one key up event so the errant key up event is still being fired.

I am responding to Freedom Scientific with this information now.

[jobara]

@jessegreenberg thanks for the update and following up with them.

[terracoda]

I agree, @jessegreenberg, great work. Looking forward to next response.

[jessegreenberg]

Thanks! I am going to remove my assignment again until we hear back from Freedom Scientific.

[jessegreenberg]

VFO responded by asking that I call the support team so that they can better understand the problem. I will do so today.

[terracoda]

@jessegreenberg, awesome :-)

[jessegreenberg]

I called the main VFO line and explained the issue. They weren't sure how to proceed during the call, but said they would discuss with their ARIA group and get back sometime tomorrow (12/16/16).

[jessegreenberg]

Got a response 12/16/16 -

Hi,
I am sending this over to our developers to have a look at.
Thanks for the report.

Removing my assignment until we hear back again.

[jessegreenberg]

VFO responded this afternoon (12/21/16). I am going to go ahead and paste the response here:

The extra keys that are sent when JAWS is running are by design since the arrow keys are attached to scripts. The user should be able to improve the situation by removing the key bindings for the jsfiddle.net domain. Here’s some guidance:

JAWS tests each keystroke to see if it is assigned to a script. If it finds that the keystroke is attached to a script, then it eats the keystroke and runs the script. If the script plays a keystroke, then the keyboard first has to be cleared so that the key can be simulated. For example, if the user presses right arrow and there is a script attached to right arrow and the right arrow script performs a right arrow, then the following sequence occurs: The right arrow down is eaten. The script is played. A right arrow up is injected to clear the keyboard. A right arrow down is injected. A right arrow up is injected. At some point during this process, the user actually releases the right arrow key and the upstroke is eaten as well.

If the keystroke is not attached to a script, then the whole process is greatly simplified. When the right arrow key is pressed, JAWS sees that the keystroke is not attached to a script and simply lets the keystroke and following upstroke through. There is no keyboard clearing or simulated keys.

So in this case, the answer may be to unbind the arrow keys for the specific application/domain. For example, create a .jkm file that is named according to the domain (http://www.freedomscientific.com/Content/Documents/Other/Domain-Scripting-Quick-Start.pdf) and in the .jkm file, add a binding to [Desktop Keys] like RightArrow=. That is, set RightArrow equal to nothing. This will unbind the keystroke when that domain is active. Note that once the user creates these bindings, the arrow keys will cease to perform their JAWS functionality. RightArrow, for example, will no longer SayNextCharacter. DownArrow will no longer SayNextLine. With this in mind, it may be useful to differentiate between numpad and extended keys (In Settings Center default.jcf, search for differentiate). With differentiation enabled, the user could keep the JAWS assignments on the numpad arrow keys while removing the bindings on the extended arrow keys.

[terracoda]

@jessegreenberg, I think we discussed these "special keys" settings in another issue a long time ago. There was something about a new JAWS specific key settings file. Not sure if I can find the issue where we discussed this, or if it is related, but this "jkm reference" sounds very familiar.

[jessegreenberg]

I also tried changing the name of the file to "www.colorado.edu.jkm", and with that change the arrow keys are working for the versions of the sim that are already deployed to phet dev.

@terracoda I remember discussing the data-at-shortcutkeys JAWS workaround, is that what you are thinking of?
https://www.freedomscientific.com/training/Surfs-Up/BypassNavigationQuickKeys.htm

[terracoda]

Yep, @jessegreenberg, I am mistaken, I was thinking of the update to JAWS 16 that uses

<div data-at-shortcutkeys={‘j’: ‘Key to navigate to next post’, ‘q’: ‘Key to place focus in chat’}></div>

Léonie Watson discusses issues around this here:
http://tink.uk/time-to-revisit-accesskey/

[terracoda]

Watson's article is a good thing to keep in mind when discuss further how to handle PhET shortcuts, in a more general sense.

[jessegreenberg]

Yes, I agree @terracoda! Watson makes great points (as usual :)

[jessegreenberg]

I responded on 12/21/16 with the following:

Thank you for this informative response, it is very helpful. I tried your suggestion and created a .jkm file for our domain and it allowed the arrow keys to work for the HTML element with the ARIA application role. However, as you mentioned in the previous email, this prevents default JAWS behavior for the arrow keys everywhere else. We do not want to disable this core method of document navigation for the entire application. Also, it is difficult to remove key bindings in this way and is not something we can ask our users to do.

Shouldn't it be the responsibility of JAWS to temporarily remove these key bindings when DOM focus is on an element that has (or whose ancestor has) the ARIA application role?

[terracoda]

@jessegreenberg, in your next response, if needed, you may want to directly quote and reference the ARIA 1.1 Standard,

When there is a need to create an element with an interaction model that is not supported by any of the WAI-ARIA widget roles, authors MAY give that element role application. And, when a user navigates into an element with role application, assistive technologies that intercept standard input events SHOULD switch to a mode that passes most or all standard input events through to the web application.

And perhaps adding a text-based explanation of an example may be useful as well. Such as,

In the case of the PhET interactive science simulations, there are many interactions that may not be supported by a native widget role, and thus would require the use of the application role. For example, there is no widget that can be used to make a 4-way drag and release for a simulation object.

[jessegreenberg]

Yes, that is a good idea @terracoda. I am going to remove assignment from this issue until we hear back from them

[emily-phet]

@jessegreenberg @terracoda - nice work folks, hopefully we hear back soon. Since it's been awhile, and through the holidays, @jessegreenberg you might want to send friendly reminder email if we don't hear back next week.

[jessegreenberg]

Good idea @emily-phet, I will reassign to me and create a self-reminder to send a friendly ping later this week.

[jessegreenberg]

I sent an email reminder to VFO on 1/17/16.

[jessegreenberg]

They responded with

Let me see what happened with this. I will be in touch soon.

[jessegreenberg]

VFO responded with:

Hi,
I have submitted your request as an enhancement request for the future.

Seems like great news! This suggests that they agree with our assessment that JAWS should be responsible removing scripts so arrow keys can be used normally with elements under the aria application role. I have no idea how they will approach this in terms of priority or what the timeline will look like.

[emily-phet]

@jessegreenberg That does seem like great news! Perhaps at the CSUN conference you can speak with them about this and get a sense of timeline.

[terracoda]

@jessegreenberg, I agree, this is great news! And a face to face discussion with a demonstration of the simulations will perhaps help create some understanding about the issue.

[emily-phet]

@jessegreenberg Can you update this issue with any information you gained from the CSUN meeting?

[jessegreenberg]

VFO now has a public bug tracker for developers. We should resubmit this report to them through that service.