New testing proposal

[vzarytovskii]

As part of our initiative of F# tests reworking, I would like to submit a proposal PR for new tests structure, frameworks, goals, and some initial prerequisites implementation.

I would really appreciate any feedback and additions.

F# Testing proposal

Why do we test

• To prevent regressions (behavioral, performance).
• To have a quicker debug feedback (thus, find problems quicker).
• To verify conformance to language spec (API contract testing).
• To have IL verification (both read and write).
• To have a quicker design feedback.
• To document behavior.

Goals

• Use one standardized testing framework across all of test projects, and get rid of custom old solutions (FSharpQA and Cambridge suites).
• Have tests restructured the way, that they are easy to discover.
• Have tests building and running on all supported platforms (Windows, macOS and Linux) and different frameworks (with exceptions when this is not applicable).
• Make it easy to run tests using standard .NET instruments (dotnet cli, test explorer, etc.).
• Leverage standard .NET testing platform and use all its benefits, suck as live unit testing, code coverage collecting, dead code elimination, etc.

Framework for testing

The following test frameworks and libraries will be used for new test projects xUnit Test Framework, FluentAssertions (+ FsUnit and FsCheck when needed). All existing NUnit test suites will be migrated to xUnit.

Justification:

• xUnit is an extensible, TDD adherent, testing framework, which was successfully adopted by many .NET engineering teams, including Roslyn, AspNetCore, EFcore, etc, has a "cleaner" approach for writing test suites (i.e. class constructor for setup, implementing IDisposable for teardown, as oppose to custom attributes). More info here.
• FluentAssertions makes it easier to write scoped assertions, provides better error messages.

Alternatives: NUnit, MSBuild, Expecto

Tests categorization

New tests should be grouped based on two factors: test type (1) + test category and subcategory (2)

1. Test type:
   Determines what type of test is it:
   • Functional tests:
     • Unit Tests: a lightweight testing for smaller modules, functions, etc.
       • Examples: Testing individual parts/functions of lexer, parser, syntax tree, standard library modules, etc.
       • Subgroups: there should be a separation between testing private and public parts of each module (i.e. compiler tests for private and public API should be in separate test projects).
     • Component Tests: testing for bigger parts of compiler.
       • Examples: Tests for the compiler components as whole, such as Code generation, IL Generation, Compiler optimizations, Type Checker, Type Providers, Conformance, etc.
     • Integration and End2End Tests: testing of F# compiler & tooling integration, as well as e2e experiences.
       • Examples: VS Integration, .NET Interactive integration, LSP integration. Integration with dotnet CLI, project creation, building, running.
   • Non-functional tests:
     • Load and Stress Tests: testing for high level modules/components to understand peak performance and potentially catch any performance regressions.
       • Examples: measuring compile, build, link times for the compiler, individual functions (i.e. data structures sorting, traversing, etc.).
2. Test category and subcategory: Tests (sub)categories shall be determined by the project, library, module, and functionality tests are covering.

Examples

• F# compiler component test which is verifying generated IL for computation expression will have category Compiler and subcategories EmittedIL and ComputationExpressions.
• F# compiler service unit test which is testing F# tokenizer, will have category Compiler.Service and subcategory Tokenizer.

Please, refer to File and project structure for more information on how tests will be organized on the filesystem.

File and project structure

Naming schema

The proposed naming schema for test projects is: FSharp.Category.Subcategory.TestType, where
Category.Subcategory is either a corresponding source project, or a more generic component (e.g. Compiler, Compiler.Private or more granular Compiler.CodeGen, Compiler.CodeGen.EmittedIL if category or subcategory project becomes too big, etc.) and TestType is the type of the test (one of UnitTests, ComponentTests, IntegrationTests, LoadTests).

Projects organization

Please refer to the "Naming schema" section above for more information on the projects naming.

New test projects will be grouped by category and test type, all subcategories are just test folders/files in the test project.

• Examples: Having test project organized like:

  tests/FSharp.Compiler.ComponentTests/CodeGen/EmittedIL/BasicTests.fs
tests/FSharp.Compiler.ComponentTests/CodeGen/StringEncoding/StringTests.fs
tests/FSharp.Compiler.ComponentTests/Optimizations/Inlining/InliningTests.fs

  Will result in one test dll "FSharp.Compiler.ComponentTests.dll" which will contain all the subcategories of tests.

• Notes:

  • This will result in reduced fragmentation of tests, all the tests files are under one big category, easier to understand what each component/unit test suite covers, less confusion in test classification for new tests.
  • If some categories (or subcategories) will become big enough - they can be factored out to a separate project.

Test Utilities/Helpers

For all new and migrated tests, any common/helper functionality shall be factored out to a separate project - FSharp.Test.Utilities.

New tests

• All new tests should be created in the new projects only.
• All new tests should contain a brief docstring description of what is being tested, link to an issue if applicable.
• All new tests should be categorized using xUnit's Trait, based on their Category and Subcategories.

Migrating existing tests

Existing FSharpQA and Cambridge need to be migrated to corresponding test projects: component-style tests to the FSharp.Compiler.ComponentTests and unittest-style tests - FSharp.Compiler.UnitTests, FSharp.Compiler.Private.Scripting.UnitTests, FSharp.Build.UnitTests, etc.

Next steps

• [In Progress] Migrate existing NUnit tests to xUnit.
• Clean up CompilerAssert.
• Make PEVerify tests work in netcore/non-windows environment.
• Start migration of existing (namely, FSharpQA and Cambridge) suites to xUnit-based projects.

Open questions:

• As far as I know, FSharp.Compiler.Service is dependant on some of the F# compiler tests. Does it have to be changed as well?

Other

Related issues: (#7075)

You can find this document under 'tests/README.md'.

[vzarytovskii]

This is WIP, some tests in CI can be failing, I've created the PR for discussions and CI testing mostly.
@ThorstenReichert @auduchinok

[Krzysztof-Cieslak]

Alternatives: NUnit, MSBuild, Expecto

Another alternative is using FsCheck also as are runner, and use it also for the example-based test. Fairly reasonable alternative, given it's already used in the code base anyway.

[forki]

can't say I'm a big fan of the xUnit change. I know it's used in C# dotnet/* but from F# perspective we had just so many spooky bugs in the ecosystem and always had to work around those things. In the end it's your decision - I personally don't think it a good one. </bikeshedding>

[vzarytovskii]

Alternatives: NUnit, MSBuild, Expecto

Another alternative is using FsCheck also as are runner, and use it also for the example-based test. Fairly reasonable alternative, given it's already used in the code base anyway.

@Krzysztof-Cieslak Thanks, Krzysztof, I will take a look at it. I personally never used it myself as a runner.
@forki Can you share what bugs did you encounter with the F# and xUnit?

[isaacabraham]

I think one of the benefits I've experienced with Expecto is the ability to make arbitrary decisions e.g. different tests for different builds without the need to jump through hoops to find different Theories and stuff. Seems like this is an issue cropping up even on the C# team.

[forki]

@vzarytovskii IIRC binding redirects issues, appdomain loading, package dependency issues, runner package issues, VS adapter issues ... - Most of those things are most likely obsolete in new shiney dotnet world, but whenever we run into issues it was very hard to communicate the problems to the project author. So I had to put xUnit work arounds into FAKE and Paket so that our users could work with xUnit. I'm not very happy about this to say the least.

[auduchinok]

As far as I know, FSharp.Compiler.Service is dependant on some of the F# compiler tests. Does it have to be changed as well?

FCS solution is also a part of this repository and shares files (including test ones) with other solutions. We work with this solution only (and it's presumably the same for some other scenarios where FCS solution is used), please update it too.

For some reason NUnit was always easier to set up for me, but I can't share any particular example. If making it use xUnit adds some considerable benefit let's change it, but currently I'm fine with FCS tests being NUnit ones.

[cartermp]

Yes, just confirming that FCS tests should also be considered here. We definitely don't want them to be orphaned. The IDE unit test suites and FCS tests cover a lot of shared concerns. Eventually it would be good to de-duplicate those concerns and have the IDE tests only cover IDE logic and have the FCS tests contain the bulk of testing specific to the compiler service.

[forki]

Is there any real need or concrete benefit to change to xunit? Or is it mainly because rest of dotnet/* is using it? I know this is bike shedding and mostly personal opinion - but from developer standpoint I'd always go expecto (or similar) + fscheck. It's just so much better to have no runner at all. It almost always wins from dev tooling aspect. But in the end I think the decision maker should be the person that maintains all this stuff. That person needs to be happy. Contributers like myself are not using it that much anyway. Phillip Carter <notifications@github.com> schrieb am Mi., 10. Juni 2020, 18:45:

…

Yes, just confirming that FCS tests should also be considered here. We definitely don't want them to be orphaned. The IDE unit test suites and FCS tests cover a lot of shared concerns. Eventually it would be good to de-duplicate those concerns and have the IDE tests only cover IDE logic and have the FCS tests contain the bulk of testing specific to the compiler service. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9418 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAOANFJ4KAODZJXUZCOX4TRV62DHANCNFSM4N2LFNVA> .

[vzarytovskii]

@forki

Is there any real need or concrete benefit to change to xunit?

Or is it mainly because rest of dotnet/* is using it?

It's this + personal preference mostly. xUnit and NUnit are frameworks I'm most familiar with.

I know this is bike shedding and mostly personal opinion - but from developer standpoint I'd always go expecto (or similar) + fscheck.

I don't have much experience using Expecto in big projects like. I don't mind using it though, if everyone agrees :)

I know @cartermp had some opinions about it.

[baronfel]

I'll say that I love expecto, but there is a persistent deadlock bug on the processing that we hit a few times a week at work with it. Recent fixes have helped, but I would take that into consideration.

[vzarytovskii]

@auduchinok

FCS solution is also a part of this repository and shares files (including test ones) with other solutions. We work with this solution only (and it's presumably the same for some other scenarios where FCS solution is used), please update it too.

Thanks, I will change it as well.

For some reason NUnit was always easier to set up for me, but I can't share any particular example. If making it use xUnit adds some considerable benefit let's change it, but currently I'm fine with FCS tests being NUnit ones.

I guess it's just more of a personal preference + adopting what other dotnet teams are using. NUnit, in my opinion, is a bit more "bloated" than xUnit.

[vzarytovskii]

@baronfel

I'll say that I love expecto, but there is a persistent deadlock bug on the processing that we hit a few times a week at work with it. Recent fixes have helped, but I would take that into consideration.

Thanks for the info!

[Krzysztof-Cieslak]

Expecto would be an interesting choice. There are... things that happened. @cartermp will know if you ask him ;-)

[vzarytovskii]

From what I've read and found about expecto, is that people are having some random hangs on bigger projects. I've never encountered this myself though.

[Krzysztof-Cieslak]

Oh, technically it's fine in my experience - I've been using it with a success in whole bunch of projects :-)

[forki]

I'm more talking about the model. Not so much about expecto itself.

Anyway most important thing is: this is now addressed systematically. Choice of test framework is not so important. Sorry for derailing the discussion.

[vzarytovskii]

Anyway most important thing is: this is now addressed systematically. Choice of test framework is not so important. Sorry for derailing the discussion.

That is absolutely fine, I really appreciate the input :) I don't just want to go, change the framework and structure, I really want to have feedback from community, so everyone is happy :)

[smoothdeveloper]

It would be interesting to see if it affects performance of running the test suite, especially in the CI environment.

[baronfel]

I would love to be able to run/debug a specific test from ionide without having to muck with VsTests' VSHOST_DEBUG environment variable to attach to the running process or whatever the arcane combination to do that is. Any solution that can do that is A-ok by me!

[vzarytovskii]

@smoothdeveloper

It would be interesting to see if it affects performance of running the test suite, especially in the CI environment.

I will for sure have some data gathered from both old and new runs to compare them.
I don’t expect much of a difference between NUnit and xUbit, however there will be some between xUnit and old Perl-based runner.

[jonsequitur]

It sounds like there's a trade-off between:

• a test framework (runner, mainly?) that just works with the .NET team's infrastructure (e.g. Arcade, dotnet test, .trx files for reporting results), and,
• a test framework (API, mainly?) that works best with idiomatic F#.

Is there a best-of-both-worlds approach? Are there improvements that we can try to make in XUnit or existing XUnit/F# bridge libraries? Also, does the shift in the F# infrastructure away from .NET Framework and toward .NET Core / .NET 5 change the calculus on anyone's opinions about the testing frameworks we've discussed?

[KevinRansom]

@forki - you asked about the switch to xunit. The ci that we use is aware of xunit testing, loging and the like. Right now we have additional scripts that smooth out some of the differences between the two frameworks within the CI. We would prefer to get rid of our special cases, and rely on the native CI behaviors.

Kevin

[KevinRansom]

@vzarytovskii

How does language testing fit into the proposed categorization scheme. A large portion of our test codebase seems to exist in-order verify that specific language constructs compile and produce executable output.

For example:
Language construct:
Dotnet Attributes
IEnumerable
For loop

Example types of tests:
Language conformance
Code generation
Symbols production and verification
Il Verification (peverify), IL producwd
Use from compiled code
Use from interactive
Interop from C#
Interop with C# implementations

I have no real intuition about what this should look like, but I would appreciate your thoughts on how we might arrange these, so that it is fairly clear where a new test belongs..

[KevinRansom]

@vzarytovskii -
forgive me if I overlooked it, but have you considered a continuous testing framework such as NCrunch, should we organize testing in order to make continuous testing usable and useful if so how would that look?

Thanks

Kevin

[vzarytovskii]

@KevinRansom

How does language testing fit into the proposed categorization scheme. A large portion of our test codebase seems to exist in-order verify that specific language constructs compile and produce executable output.

I have no real intuition about what this should look like, but I would appreciate your thoughts on how we might arrange these, so that it is fairly clear where a new test belongs..

My idea is to organize the compiler tests into separate suite(s) of component tests which will be testing certain categories (e.g. codegen, IL r/w, interop, etc.).
The first step is to start moving out of cambridge & FSharpQA suites towards separate xUnit ones, so we can have it runnable/debugable on the standard test platform.

forgive me if I overlooked it, but have you considered a continuous testing framework such as NCrunch, should we organize testing in order to make continuous testing usable and useful if so how would that look?

To be honest, I did not even consider using NCrunch at this point, due to its licensing (correct me if I'm wrong, but I thought that NC is commercial software, with no OSS license). And I'm unaware of any suitable alternatives which would support both desktop and coreclr.

[forki]

From my experience with ncrunch I would not make that a priority. It's an awesome tool and seeing those red and green bubbles along the covered code is amazing. But it's really hard to setup properly and to maintain it. From my experience you get most of the same with just expecto style testing and dotnet watch run. You don't have the code coverage, but you do have that constant feedback loop. Which is the important part IMHO Vlad Zarytovskii <notifications@github.com> schrieb am Do., 11. Juni 2020, 10:55:

…

@KevinRansom <https://github.com/KevinRansom> How does language testing fit into the proposed categorization scheme. A large portion of our test codebase seems to exist in-order verify that specific language constructs compile and produce executable output. I have no real intuition about what this should look like, but I would appreciate your thoughts on how we might arrange these, so that it is fairly clear where a new test belongs.. My idea is to organize the compiler tests into separate suite(s) of component tests which will be testing certain categories (e.g. codegen, IL r/w, interop, etc.). The first step is to start moving out of cambridge & FSharpQA suites towards separate xUnit ones, so we can have it runnable/debugable on the standard test platform. forgive me if I overlooked it, but have you considered a continuous testing framework such as NCrunch, should we organize testing in order to make continuous testing usable and useful if so how would that look? To be honest, I did not even consider using NCrunch at this point, due to its licensing (correct me if I'm wrong, but I thought that NC is commercial software, with no OSS license). And I'm unaware of any suitable alternatives which would support both desktop and coreclr. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#9418 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAOANHES2YGE2EFZ5UIRJDRWCLZNANCNFSM4N2LFNVA> .

[KevinRansom]

@vzarytovskii -- As I understand it, there are developers who like to use it as part of their workflow. Have we considered, whether the changes we make would suit such a workflow?
@colombod and @jonsequitur are both big fans, so we could start by seeing if they have any advice on how tests should be organized, written and deployed to make continuous testing usefull.

Kevin

[vzarytovskii]

@KevinRansom

@vzarytovskii -- As I understand it, there are developers who like to use it as part of their workflow. Have we considered, whether the changes we make would suit such a workflow?

As far as I know, NCrunch is compatible with all modern frameworks and I expect it (and live unittesting) work just fine with xUnit. In fact, after the initial migration it will be a separate goal - to enable these sorts of scenarios for our tests.

[TIHan]

We should just simply use xUnit and that be our only dependency. At the moment, I don't see value with FluentAssertions in the context of testing compilations; most of the assert/verification calls for testing compilations are abstractions that happen to use NUnit / xUnit assertions as an implementation detail. But when testing FSharp.Core functionality, it might be a benefit.

If we find that using other libraries would benefit us, we should discuss about bringing them in when the time comes. I know NCrunch is on the table, but I would not make it a priority.

Also, I strongly encourage everyone to please look at how the Roslyn compiler tests are done. IMO, they make it very simple.

[TIHan]

Out of curiosity, is FluentAssertions giving benefit to any of the current tests?

[vzarytovskii]

I guess only a bit better assertion messages. They seem more readable to me.

[jonsequitur]

Enabling NCrunch doesn't require that we do anything special in the repo. It's not a library dependency. In practice it works reliably until you overcustomize a build. Keeping the build simple has other virtues anyway, so I think it's worthwhile to keep this kind of workflow within reach.

[vzarytovskii]

I would agree with @TIHan here. We should start with plain simple xUnit here and then add other dependencies when needed as we migrate more tests and scenarios.

[zpodlovics]

Would it be possible to replace the printf "magic" with some simple formatting functions on the top during the migration ? To avoid issues like this: #4954 as it would help improve the F# test coverage on platforms / runtimes / targets like Mono AOT, WebAssembly, Fable, Fez, etc.

[vzarytovskii]

@zpodlovics

Would it be possible to replace the printf "magic" with some simple formatting functions on the top during the migration ? To avoid issues like this: #4954 as it would help improve the F# test coverage on platforms / runtimes / targets like Mono AOT, WebAssembly, Fable, Fez, etc.

Hm, great feedback on tests, thanks. I will need to look at it more thoroughly though.

Most likely, won’t be a part of this particular PR.

[KevinRansom]

This looks like a great start.

[KevinRansom]

Should the test files be all in the same directory ByteMemoryTests.fs, EditDistance.fs, HashIfExpression.fs. or should they be further differentiated, it's just that I imagine eventually Compiler.ComponentTest would have a huge number of files in it.

[vzarytovskii]

@KevinRansom my idea was grouping files by the directories inside the test suites (e.g. Compiler.ComponentTests). But they can as well be factored out into separate suite if will grow in size.