Separate test runner API

[connor4312]

From discussion offline:

@jrieken: I still haven’t started nit-mode but things are looking much better. One thing that occurred to me is about createTestRun. Its comment says that a label can be used to distinguish different platforms on which the test run. I wonder if that’s enough? Not all tests run on all platforms (like for us only things from /browser/ or below run in safari) and it doesn’t offer a way for users to say “Re-run/run in Firefox”. Is it that a test run must always occur on all platforms? Should TestItem.runnable list “test runners”? What’s your take on this? Has this been already discussed and decided?
@connor4312: This is something the playwright team brought up as well, more around wanting to allow users to select what browsers they want to run on. Perhaps we can introduce a new "configurations" concept, where there is a set-like object of them on controllers. The UI could provide a way to run subsets of configurations (or the entire list of configurations) which could be passed in the test run request. Wrt items that only run on one platform, I think the way to do those right now would be marking them as "skipped" when run on platforms where they aren't valid. Not sure I want to add a bunch of complexity to the current "runnable" boolean...

Perhaps this could be something like this:

export class TestRunner<T = any> {
	name: string;
	runHandler: (request: TestRunRequest<T>) => void;
	constructor(name: string, runHandler: (request: TestRunRequest<T>) => void);
}

/**
 * Interface to discover and execute tests.
 */
export interface TestController<T = any> {
	registerTestRunner(runner: TestRunner<T>): Disposable;

With explicit handlers, this calls into question the need to do the magic matching on the request in createTestRun. However there is still the case of createTestRun being used to publish test runs without a run actually being executed in the vscode UI.

[connor4312]

So I realized what I want to express in the UI is not runners, but configurations. For example, in playwright you might run tests in edge, run tests in all browsers, or do this but with coverage, and so on. This leads me to a different design, where explicitly allow extensions to create configurations:

// Todo: this is basically the same as the TaskGroup, which is a class that
// allows custom groups to be created. However I don't anticipate having any
// UI for that, so enum for now?
export enum TestRunConfigurationGroup {
	Run = 1,
	Debug = 2,
	Coverage = 3,
}


export interface TestRunConfiguration {
	/**
	 * Configures where this configuration is grouped in the UI. If there
	 * are no configurations for a group, it will not be available in the UI.
	 */
	readonly group: TestRunConfigurationGroup;
	
	/**
	 * Label shown to the user in the UI.
	 */
	label: string;

	/**
	 * Controls whether this configuration is the default action that will
	 * be taken when its group is actions. For example, if the user clicks
	 * the generic "run all" button, then the default configuration for
	 * {@link TestRunConfigurationGroup.Run} will be executed.
	 */
	isDefault: boolean;

	/**
	 * If this method is present a configuration gear will be present in the
	 * UI, and this method will be invoked when it's clicked. When called,
	 * you can take other editor actions, such as showing a quick pick or
	 * opening a configuration file.
	 */
	configureHandler?: () => void;

	/** <moved from the TestController> */
	runHandler: (req: TestRunRequest, token: CancellationToken) => void;

	/**
	 * Deletes the run configuration.
	 */
	dispose(): void;
}

export interface TestController {
	createRunConfiguration(label: string, group: TestRunConfigurationGroup, runHandler: (req: TestRunRequest) => void): TestRunConfiguration;

In this case playwright could, for example, have a "run all" configuration, which creates multiple runs when invoked, in addition to individual browsers. The TestRunRequest can then omit the debug boolean, since it would use these configurations instead.

And we can also remove debuggable and runnable from the TestItem. That introduces a gap for Go/Python "subtests" where children are not runnable, but I think in these cases it would be acceptable for them to automatically run the parent of the requested subtest. Debugability would be indicated by the presence of any Debug-group configuration.

With this we would can keep the the current execution model unchanged, aside from the adjustment to the TestRunRequest and location of the runHandler. I think this approach solves a lot of problems.

[jrieken]

So I realized what I want to express in the UI is not runners, but configurations. For example, in playwright you might run tests in edge, run tests in all browsers, or do this but with coverage, and so on

I like that thinking. I'd say the controller is the runner (without suggesting to change the name) and configs are all different ways to make the runner "do stuff"

[connor4312]

@connectdotz mentioned auto run, which is a useful UI utility but is a bit of a sore spot, since currently VS Code core implements auto run but extensions can potentially do it better themselves.

I suggest an additional method on the configuration:

export interface TestRunConfiguration {
    // called with true when auto run is requested, false when it's turned off.
    watchHandler?: (enabled: boolean) => void;

[connectdotz]

@connor4312 if we are going to use the configuration handlers as an indicator for the UI, a few questions:

1. in multi-root projects, they might have different configurations per workspace folder, how should these be handled? Right now I used the TestItem level runnable to control on the item level, if we are going to roll them up to the configuration level per extension, we might end up dealing with lots of "sorry this action is not supported for these tests" post messages or unnecessary SKIPPED state that blocked the real run state...
2. For multi-root with mixed autoRun situations, how should the UI experience be like? Do we keep the button active and when invoking the onDidChangeAutoRun we show warning/error messages? How will the user know which workspace is under autoRun watch if the autoRun switch on top does not necessarily reflect the true state?

I don't object to the configuration-driven run API, but maybe we could still keep the item level config so they can express the micro-state/action like runnable, debuggable, and maybe even autoRun state?