From 79ec57969594c5985fbefc44ed1188583b5e79c2 Mon Sep 17 00:00:00 2001 From: jugglinmike Date: Mon, 23 Sep 2024 19:56:37 -0400 Subject: [PATCH] Refactor "pressKeys" as one of many "user intents" (#76) In conversation with assistive technology vendors, we have learned that automating arbitrary key presses does not necessarily align with the security model of all platforms. Separately, we anticipate introducing means to simulate more specific types of user interaction (e.g. "move to next heading"). Because these may themselves be implemented at the vendor's discretion (and because we wish to allow for vendors to experiment with the development of still more kinds of interaction), the space for simulating interaction may be somewhat fragmented, particularly in the initial deployments of this protocol. We have decided to design a command which accommodates this variability so that the presence/absence of interactions can be communicated consistently (note the "unknown user intent" error) and so that additional interactions can be introduced without the addition of entirely new commands (or entirely new extension commands, as the case may be). --- index.bs | 102 +++++++++++++++++++++++++--------- schemas/at-driver-remote.cddl | 22 ++++++-- 2 files changed, 91 insertions(+), 33 deletions(-) diff --git a/index.bs b/index.bs index 7c9df48..90e054b 100644 --- a/index.bs +++ b/index.bs @@ -337,6 +337,10 @@ The following table lists each error code, its associated JSON `error session not created `session not created` A new [=session=] could not be created. + + unknown user intent + `unknown user intent` + The remote end does not support a user intent with the provided name. cannot simulate keyboard interaction `cannot simulate keyboard interaction` @@ -824,12 +828,53 @@ Issue: Do we need a "setting changed" event? The Interaction Module {#module-interaction} -------------------------------------------- +The following table of standard user intents enumerates the user intents each implementation must support. + + + + + +
Standard user intents
Name + Algorithm +
"`pressKeys`" + [=press keys=] +
+ +A [=remote end=] has a table of extension user intents, which is a mapping of zero or more string names of user intents and algorithms for simulating the named intents. Extension user intents' names must contain a "`:`" (colon) character, denoting an implementation specific namespace. + +
+ +Note: Each string in `KeyCombination` represents a "raw key" consisting of a +single code point with the same meaning as in WebDriver's keyboard +actions. For example, `["\uE008", "a"]` means holding the left shift key +and pressing "a", and then releasing the left shift key. [[WEBDRIVER]] + +Issue(34): This algorithm does not yet have a means for indicating a screen-reader specific modifier key (or keys). + +Issue(51): This algorithm only supports one specific kind of press/release sequence, and it is not clear if that is sufficient to express all keyboard commands in all implementations. + +To press keys given |command parameters|: + +1. [=Try=] to [=check that keyboard interaction can be simulated=]. +2. [=Try=] to [=check that one of the expected applications has focus=]. +3. Let |keys| be the value of the keys field of |command + parameters|. +4. [=list/For each=] |key| of |keys|: + 1. Run [=implementation-defined=] steps to simulate depressing |key|. +5. [=list/For each=] |key| of |keys| in reverse [=List=] order: + 1. Run [=implementation-defined=] steps to simulate releasing |key|. +6. Let |body| be a new [=map=]. +7. Return [=success=] with data |body|. + +
+ ### Definition ### {#module-interaction-definition} [=Remote end definition=]: -InteractionCommand = (InteractionPressKeysCommand) +InteractionCommand = (InteractionUserIntentCommand) [=Local end definition=]: @@ -851,32 +896,38 @@ InteractionCapturedOutputParameters = { ### Commands ### {#module-interaction-commands} -#### The interaction.pressKeys Command #### {#module-interaction-presskeys} - -The interaction.pressKeys command simulates pressing a key combination on a keyboard. +#### The interaction.userIntent Command #### {#module-interaction-userintent} -Issue(34): This command does not yet have a means for indicating a screen-reader specific modifier key (or keys). - -Issue(51): This algorithm only supports one specific kind of press/release sequence, and it is not clear if that is sufficient to express all keyboard commands in all implementations. +The interaction.userIntent command simulates pressing a key combination on a keyboard.
[=Command Type=] 
-  InteractionPressKeysCommand = {
-    method: "interaction.pressKeys",
-    params: InteractionPressKeysParameters
+  InteractionUserIntentCommand = {
+    method: "interaction.userIntent",
+    params: InteractionUserIntentParameters
   }
 
-  InteractionPressKeysParameters = {
+  InteractionUserIntentParameters = (
+    PressKeysIntentParameters /
+    ExtensionIntentParameters
+  )
+
+  PressKeysIntentParameters = {
+    "name" => "pressKeys",
     "keys" => KeyCombination,
-    Extensible,
   }
 
   KeyCombination = [
     1* text
   ]
+
+  ExtensionIntentParameters = {
+    "name" => text,
+    Extensible,
+  }
Result Type @@ -888,24 +939,21 @@ Issue(51): This algorithm only supports one specific kind of press/release seque
-Note: Each string in `KeyCombination` represents a "raw key" consisting of a -single code point with the same meaning as in WebDriver's keyboard -actions. For example, `["\uE008", "a"]` means holding the left shift key -and pressing "a", and then releasing the left shift key. [[WEBDRIVER]] +
-The [=remote end steps=] given |session| and |command parameters| are: +The [=remote end steps=] given session and |command parameters| are: -1. [=Try=] to [=check that keyboard interaction can be simulated=]. -2. [=Try=] to [=check that one of the expected applications has focus=]. -3. Let |keys| be the value of the keys field of |command +1. Let |name| be the value of the name field of |command parameters|. -4. [=list/For each=] |key| of |keys|: - 1. Run [=implementation-defined=] steps to simulate depressing |key|. -5. [=list/For each=] |key| of |keys| in reverse [=List=] order: - 1. Run [=implementation-defined=] steps to simulate releasing |key|. -6. Let |body| be a new [=map=]. -7. Return [=success=] with data |body|. +2. If there is an entry in the [=table of standard user intents=] with name |name|: + 1. Let |algorithm| be the algorithm associated with user intent |name| in the [=table of standard user intents=]. +3. Otherwise, if there is an entry in the [=table of extension user intents=] with name |name|: + 1. Let |algorithm| be the algorithm associated with user intent |name| in the [=table of extension user intents=]. +4. Otherwise: + 1. Return an [=error=] with [=error code=] [=unknown user intent=]. +5. Return the result of evaluating |algorithm| given |command parameters|. + +
### Events ### {#module-interaction-events} diff --git a/schemas/at-driver-remote.cddl b/schemas/at-driver-remote.cddl index 94e3cd0..efc48c0 100644 --- a/schemas/at-driver-remote.cddl +++ b/schemas/at-driver-remote.cddl @@ -74,18 +74,28 @@ SettingsGetSupportedSettingsCommand = { params: EmptyParams } -InteractionCommand = (InteractionPressKeysCommand) +InteractionCommand = (InteractionUserIntentCommand) -InteractionPressKeysCommand = { - method: "interaction.pressKeys", - params: InteractionPressKeysParameters +InteractionUserIntentCommand = { + method: "interaction.userIntent", + params: InteractionUserIntentParameters } -InteractionPressKeysParameters = { +InteractionUserIntentParameters = ( + PressKeysIntentParameters / + ExtensionIntentParameters +) + +PressKeysIntentParameters = { + "name" => "pressKeys", "keys" => KeyCombination, - Extensible, } KeyCombination = [ 1* text ] + +ExtensionIntentParameters = { + "name" => text, + Extensible, +}