adds subset of timeout options for harness

[ChrisC]

I suspect that using a higher timeout after starting up Safari in CI may resolve some issues we've seen where the VO Bot seems to navigate to the wrong place, like in these issues:

• VoiceOver Bot navigates to incorrect element early in test runs w3c/aria-at-app#1294
• Voiceover Bot + Firefox - Bot navigates into MacOS system menu in tests/menu-button-actions-active-descendant w3c/aria-at-app#1291
• Possibly even VoiceOver Bot / any browser - Expected button states are inconsistent in Disclosure Nav tests w3c/aria-at-app#1298

Using higher timeouts when the harness locally also seems to resolve some of these issues. If this does make a difference after running a new consistency report, we may actually just want to update some of the timeout defaults on the harness itself.

[gnarf]

I'm not seeing anything consuming the timeout params to the github actions - is this missing a change to the .sh file?

Have you run the consistency check against this branch? Did it show any improvement?

[ChrisC]

I'm not seeing anything consuming the timeout params to the github actions - is this missing a change to the .sh file?

Have you run the consistency check against this branch? Did it show any improvement?

Aw missed the .sh file! And good tip to try running the check with this branch to test!

[jugglinmike]

Thanks, Chris!

We need one extra bit of configuration to translate the "Workflow input" values to the environment variables that are referenced in the Bash script. I couldn't describe them with a suggestion, so I just pushed a commit to add them. Hope that's okay!

Separately, it makes sense that we aren't defining these workflow "inputs" for the Windows Workflow, but we are specifying them regardless. GitHub's docs don't make it clear what happens if a Workflow is invoked with unrecognized inputs, though. It seems likely that they would just ignore them, but it might be worth doing a test run just to be sure. I'll let you decide about that, so I won't merge this myself.

[jugglinmike]

Almost forgot to specify default values for the values (since this patch is structured to specify the arguments in all cases, the built-in CLI defaults won't be used)

[ChrisC]

Separately, it makes sense that we aren't defining these workflow "inputs" for the Windows Workflow, but we are specifying them regardless. GitHub's docs don't make it clear what happens if a Workflow is invoked with unrecognized inputs, though. It seems likely that they would just ignore them, but it might be worth doing a test run just to be sure. I'll let you decide about that, so I won't merge this myself.

It turns out that the windows scripts were erroring out without the inputs being defined in the Windows workflow as you suspected @jugglinmike . I actually had to do a bit of debugging to get the input value types to work (sorry for all the extra commits here).

I also ran some CI tests, and while introducing these delays does seem to improve VO consistency... it's not as a big difference as I had hoped, not does it seem to address the issues with incorrect navigation or inconsistent buttons states listed in the description ☝🏽 😞

I ran this branch with the disclosure-navigation tests. We do see some incremental improvements when adding more timeout delays, but nothing dramatic. For example, VO inconsistencies on Firefox in that test plan dropped from 25% in the last full run to ~14-17% in recent runs with the "After Navigation" timeout increased to 2 seconds (default is 1 second). Oddly, in a run with an increased "Doc Ready" timeout increased to 5 seconds (default is 2 seconds), Safari actually did worse than the Jan run with the default timeouts.

I'm actually inclined to add the rest of the timeout variables as possible inputs to give us more levers to experiment with in the future, but it's not looking like these timeouts will close any of the reported VO bugs so far :-( I would also be open to just closing this PR and continue to look for root causes of the bugs listed above.

What do you all think @gnarf @jugglinmike ?