[Testing Platform] Batch Filter

[thomhurst]

This would work alongside #3527.

A filter type that could specify a start index and an end index, and pass that filter through to test adapters.

Test adapters could implement this and skip/batch as appropriate. All the testing platform needs to do is pass through a new filter type with the start/end indexes on it.

This would enable users/build systems to run batches of tests on different processes/agents to speed up test runs using parallelism.

Logic could be as follows:

--dotnet run --count-tests

500

dotnet run --batch-filter 0 99
dotnet run --batch-filter 100 199
dotnet run --batch-filter 200 299
dotnet run --batch-filter 300 399
dotnet run --batch-filter 400 499

public sealed class BatchFilter : ITestExecutionFilter
{
    public required int StartIndex { get; init; }
    public required int EndIndex { get; init; }
}

[Evangelink]

This is assuming that the tests are always returned in the same order right?

I don't see a direct big value in this shape (I would prefer to split based on list of UIDs) but I don't think that's a bad suggestion so I'll mark it as feature request and see if we have some more interest before we work on implementation.

[thomhurst]

Yes you're right they'd need to be in the same order. But if framework providers register it like they do your treenode filter, they can choose whether to support it or not.

My tests are source generated so should be deterministic afaik

[thomhurst]

Splitting on uids works too. Wouldn't require specific ordering then, but would be a very verbose dotnet run command

[thomhurst]

Also are you planning to let frameworks build their own filters? That's all still internal code isn't it?

[Evangelink]

but would be a very verbose dotnet run command

Yes, my idea was to be able to provide a filter file with either simply the list of uids OR to have a structured file so we are not only expecting UIDs but we could potentially do mix up of filters.

Also are you planning to let frameworks build their own filters? That's all still internal code isn't it?

Yes and yes. We could start opening with the [Experimental]. While we do that, I think you should be able to implement some custom with "hacky" solution. Register an extension that will provide this command line and when executed simply stores the info in the services, from your test framework, retrieve this custom service and get the data (e.g. start/end index) and voila.

cc @MarcoRossignoli

[thomhurst]

I was actually thinking about mixes of filters earlier and whether that was worth supporting. You'd need a new type like AggregateFilter (like AggregateException) that wraps around multiple. Might be worth splitting that into its own issue if that sounds like a good idea to you?

[Evangelink]

Yes definitely!

[MarcoRossignoli]

I think that to have complete filtering we should offer

• precise set of tests UIDs
• globbing patterns

Users can OR them, treenode is already supporting the globbing part we should add something like --filterByUid-iniline "uid uid uid" --filterByUid-fromfile filtersIds.txt (where the filtersIds.txt is a file where we have 1 id per line).

[thomhurst]

You already have TestNodeUidListFilter but I don't know how to invoke it from the cli. I assume it's to support test explorers mainly ATM.

[MarcoRossignoli]

You already have TestNodeUidListFilter but I don't know how to invoke it from the cli. I assume it's to support test explorers mainly ATM.

Yep it's only used by the protocol atm. If we expose the api in the cli we can fill that object too.