Run Windows tests by default

[pjanotti]

Component(s)

No response

Describe the issue you're reporting

As of August of 2024 Windows tests are run only when the label Run Windows is added to a PR. The result is that many PRs, that in principle are not specific to Windows, end up being merged without running tests on Windows. Due to subtle differences between the OSes (scheduling, file systems, etc) we end up with bugs and flaky Windows tests. These are only caught in unrelated PRs (that have the label) or after merge when Windows tests are ran on main.

This issue is to track the eventual deprecation of the Run Windows label, i.e.: in the end all tests should run by default on Windows.

I will keep updating this top issue if identified issues and corresponding fixes. This is expected to take some time.

[crobert-1]

I'm concerned about the potential impact of running Windows tests on every PR. I know as it is the contrib repo puts a pretty heavy burden on CNCF's GitHub runners, and this would just add to it.

I don't have a ton of context here though, others would likely have more informed input. It's not a blocker for me, just wanted to call it out.

[pjanotti]

That's a good point @crobert-1 - in general it would be better if we just ran the tests of the affected components, that said this seems a separate issue that can be handled concurrently to this one and even have it as a pre-requisite to finally enabling Windows tests on every PR.

[pjanotti]

I will keep monitoring the runs, but, it seems that they are in much better shape right now.