Support for continuous (autorun) tests

[connor4312]

A popular feature in test runners is re-running tests automatically as source code changes. I this should consider recognizing API, since the UI will have several specific behaviors for this:

• A profile-aware "toggle" button to start and stop running common across extensions and implementations
• Different behavior around when or if the peak automatically opens and state is followed in the test explorer when under an auto-run

While the former can be done through existing extension APIs such as menu contributions, it may be worth having official UI for this to provide a consistent experience, especially when multiple types of tests (and therefore extensions) are present.

I think this can be implemented in a fairly straightforward way with the addition of a supportsContinuousRun on the profile, and a boolean indicator on TestRuns indicating whether they should be continuous. I prefer the term "continuous" over "auto run" since saying "this is a continuous test run" brings in the temporal nature that "this is an autorun" does not.

API Proposal

	export interface TestRunProfile {
		/**
		 * Whether this profile supports continuous running of requests. If so,
		 * then {@link TestRunRequest.continuous} may be set to `true`. Defaults
		 * to false.
		 */
		supportsContinuousRun: boolean;

		/**
		 * Handler called to start a test run. When invoked, the function should call
		 * {@link TestController.createTestRun} at least once, and all test runs
		 * associated with the request should be created before the function returns
		 * or the returned promise is resolved.
		 *
		 * If {@link supportsContinuousRun} is set, then {@link TestRunRequest2.continuous}
		 * may be `true`. In this case, the profile should observe changes to
		 * source code and create new test runs by calling {@link TestController.createTestRun},
		 * until the cancellation is requested on the `token`.
		 *
		 * @param request Request information for the test run.
		 * @param cancellationToken Token that signals the used asked to abort the
		 * test run. If cancellation is requested on this token, all {@link TestRun}
		 * instances associated with the request will be
		 * automatically cancelled as well.
		 */
		runHandler: (request: TestRunRequest, token: CancellationToken) => Thenable<void> | void;

		// ...
	}

	export interface TestController {
		/**
		 * Creates a profile used for running tests. Extensions must create
		 * at least one profile in order for tests to be run.
		 * @param label A human-readable label for this profile.
		 * @param kind Configures what kind of execution this profile manages.
		 * @param runHandler Function called to start a test run.
		 * @param isDefault Whether this is the default action for its kind.
		 * @param tag Profile test tag.
		 * @param supportsContinuousRun Whether the profile supports continuous running.
		 * @returns An instance of a {@link TestRunProfile}, which is automatically
		 * associated with this controller.
		 */
		createRunProfile(label: string, kind: TestRunProfileKind, runHandler: (request: TestRunRequest, token: CancellationToken) => Thenable<void> | void, isDefault?: boolean, tag?: TestTag, supportsContinuousRun?: boolean): TestRunProfile;

		// ...
	}

	export class TestRunRequest {
		/**
		 * Whether the profile should run continuously as source code changes. Only
		 * relevant for profiles that set {@link TestRunProfile.supportsContinuousRun}.
		 */
		readonly continuous?: boolean;

		/**
		 * @param tests Array of specific tests to run, or undefined to run all tests
		 * @param exclude An array of tests to exclude from the run.
		 * @param profile The run profile used for this request.
		 * @param continuous Whether to run tests continuously as source changes.
		 */
		constructor(include?: readonly TestItem[], exclude?: readonly TestItem[], profile?: TestRunProfile, continuous?: boolean);

		// ...
	}

[connor4312]

cc @smcenlly @karthiknadig

[JustinGrote]

For Pester Tests, currently I control this with a AutoRun setting, so it makes perfect sense to put it in profiles.

One issue that has come up is, when autorunning, it is possible to save a file and commit with the test pane and test file closed, and have no idea what you changed just failed the test, there's no indicator in this situation

Thoughts:

1. If the ability to "alias" a test e.g. put a run button next to the function it references, and have that update, that would provide a good indicator, but still not useful if the user scrolls away after saving and doesn't notice. Maybe some sort of coverage API integration.
2. Maybe allow a configurable setting to report test errors to the "Problems" pane.
3. Have a new editor tab indicator similar to the git "modified/error/etc." indicators that specifically notes a test failed here

[ashgti]

With this API, who is responsible for detecting test run trigger events? Is the implication that the extension should run tests on FS save or on editor text change?

Also, what is the flow of the TestRunProfile.runRequest, does it get called one time when continuous mode is enabled and then the cancellation token is cancelled when the mode is disabled or does it get called once per document save event?

[connor4312]

Is the implication that the extension should run tests on FS save or on editor text change?

That's correct

Also, what is the flow of the TestRunProfile.runRequest, does it get called one time when continuous mode is enabled and then the cancellation token is cancelled when the mode is disabled or does it get called once per document save event?

Once when enabled

[JDaance]

I'm looking at implementing auto run right now and have a bit of trouble figuring out the best way. As it stands now, should I create a new run profile called continous or something, and simply never "return"? Then it can be cancelled by cancellation token?

Edit: And for individual tests that I want to autorun I need to contribute a menu item to start such a run?

This might work! I did like using test explorer ui, but it was difficult to code the auto run without duplicate runs since the extension and my adapter where fighting about whos in control.

[connor4312]

You still want to call TestController.createTestRun whenever you start running tests -- on file save, for example. Note that you can just new your own TestRunRequest, it doesn't have to come from a runHandler invokation. Then use existing contribution points to add your own auto run toggle UI, for example a status bar contribution.

[JDaance]

You still want to call TestController.createTestRun whenever you start running tests -- on file save, for example. Note that you can just new your own TestRunRequest, it doesn't have to come from a runHandler invokation. Then use existing contribution points to add your own auto run toggle UI, for example a status bar contribution.

Ok gotcha, it would be nice to have the auto run toggle in the test explorer somewhere, but I guess thats what this issue is about.

I am a bit confused about the settings available for the testing feature concerning auto run. I guess that these settings currently do not have any effect?

Edit: It would also be nice to have the auto run flag per-test as in the old UI, I did use that a bit.

[connor4312]

I guess that these settings currently do not have any effect?

That's right

[JDaance]

Poor mans autorun toggle, until better support is available :)

Turns out I can change the label if I keep the reference

[JustinGrote]

I also implemented a similar item in Pester Tests that all centers around toggling a setting.
pester/vscode-adapter#96

[dogweather]

I guess that these settings currently do not have any effect?

That's right

Could we hide these features that fail silently until auto-run ships?

[B1773rm4n]

How often this will get delayed again?

[matepek]

Hello @connor4312 ,

If I understand correctly one can just select a run profile to "autorun" but cannot select a set of tests.
We are hoping it will be supported in the future but I'm not sure how the current api supports that.
Maybe request: vscode.TestRunRequest2 can return with the included tests..
And then we just have to check that the modified file is relates to the included TestItems or not.

What do you think?

[connor4312]

vscode.TestRunRequest2 extends vscode.TestRunRequest, so also has "include" files. I just haven't added UI for that yet. Thanks for letting me know it's something you're interested in 🙂

[JustinGrote]

vscode.TestRunRequest2 extends vscode.TestRunRequest, so also has "include" files. I just haven't added UI for that yet. Thanks for letting me know it's something you're interested in 🙂

Yes I imagine for larger projects it would be nice to autorun at a particular scope to get "fast feedback" without waiting for all the other tests to run.

[ioquatix]

I was under the impression that it might be possible to provide per-test coverage, so that we know what tests to run when a specific line of code is changed.

[connor4312]

I was under the impression that it might be possible to provide per-test coverage, so that we know what tests to run when a specific line of code is changed.

This is up to extensions / test runners to provide. VS Code itself is not aware of your dependency tree or runtime environment. If the test runner you're working with generates per test coverage, that could be a reasonable way to figure out how to map code to tests.

[matepek]

I think I found an issue: debugging my extension with the insider the "Test Output" is only:

The test run did not record any output.

instead of the one I generated.

[ioquatix]

I've been playing around with triggering a test run when files are modified.

This UI pops up while I'm editing the test, which can be clunky:

It would be good to think about a way to make sure the UI and test feedback is compatible with real-time editing of the test.

[connor4312]

Thanks, will look at that.

I have been out of office this month, so I will look to finalize this API (and clean up polish items as mentioned) in March

[ioquatix]

If you'd like me to walk through how my particular integration is going, and we can check the weak points and strong points, let me know :)